Piping code

[Bifrozt]

Discovered a "new" way in which an attacker can transfer files to the honeypot without HonSSH being able to detect or intercept the file.
This issue was only detected when the IDS started sending alerts about outbound SSH attacks.

The instance was not configured to send any email alerts, so i figured that someone had compromised the honeypot and downloaded some malware but, there was no entries in the downloads.log or any downloads directory in the sessions file structure. The only obvious thing was a EXEC session log, these are the entries that show the activity of the attacker

20160214_122650_425601 - [EXEC0] Opened Channel
20160214_122650_426107 - [EXEC0] Command Executed: mkdir /tmp/.xs/
20160214_122651_364395 - [EXEC0] Closed Channel
20160214_122651_647226 - [EXEC1] Opened Channel
20160214_122651_647703 - [EXEC1] Command Executed: cat >/tmp/.xs/daemon.i686.mod
20160214_122714_097971 - [EXEC1] Closed Channel
20160214_122714_641976 - [EXEC2] Opened Channel
20160214_122714_642469 - [EXEC2] Command Executed: chmod 777 /tmp/.xs/daemon.i686.mod
20160214_122715_559964 - [EXEC2] Closed Channel
20160214_122715_836329 - [EXEC3] Opened Channel
20160214_122715_836800 - [EXEC3] Command Executed: /tmp/.xs/daemon.i686.mod
20160214_122716_420530 - [EXEC3] Closed Channel
20160214_122716_421130 - [SSH  ] Lost Connection with 1.2.3.4

The only thing that raised a red flag was the size of the adv-log's, some of them were larger than 3 MB, after digging trough them i discovered the following strings

• \xeaUPX!\xd8
  UPX
• \x7fELF\x01
  ELF

Given the size of the adv-log's and the presence of these strings we can assume that the attacker used SSH to move binary files to the honeypot.

After a couple of tests i think i figured out how the attacker did this.
It's possible to pipe things from the local host into a SSH EXEC session.

echo "Test Test Test" | ssh -l derp 10.13.111.57 'cat - > /tmp/pipe_in_exec.txt'

This will pipe the output from the echo over SSH to /tmp/pipe_in_exec.txt on 10.13.111.57. Given that this is possible, might it be that the attacker used a similar technique to transfer the file?

Is this something that should be within the scope of HonSSH to catch/intercept/detect?

[robputt]

What we could really do with is some tasty inotify stuff to log any file changes on the Honeypot and attribute in a log somewhere file changes that occurred during the session time period. Only problem is feeding these events to the HonSSH server is likely to require some agent or similar which could of course be detectable and potentially interesting to Mr Hacker. Having said that most of them run some automated payload retrieval and execution script so maybe they wouldn't notice it...

[Bifrozt]

Yes, its possible to install some type of monitoring software to do this.
Like using OSSEC((http://ossec.github.io/index.html) to monitor critical directories and log files, have the OSSEC agent to report everything to a centralized server like OSSIM, deploy an IDS that's monitoring the network traffic and have everything correlated and analyzed to trigger alerts....But i doubt that everyone wanna run and maintain a setup like this 😄

You are correct that the majority of the automated attacks never bother to look for monitoring agents or hidden kernel modules. The attackers that are looking for this type of software is the interesting ones. All of this is besides the point tho.

The fact is that its possible to transfer files from a remote host to the honeypot, while using HonSSH as the proxy. It's probably possible to mitigate by setting disable_exec = true.

[bang-uin]

If one is using docker, inotify could solve all the problems regarding honssh not being able to detect some file download mechanisms.

AUFS storage driver:
For each started container docker will maintain the rootfs diff of its read/write layer under /var/lib/docker/aufs/diff.

Starting with docker 1.10 the container id does not correspond to directory name anymore. But this one liner $(cat /var/lib/docker/image/aufs/layerdb/mounts/$(docker inspect --format {{.Id}} <YOUR_CONTAINER_NAME>)/mount-id) will tell what directory is used.

Using this combined with inotify would give everything needed to copy away modified and created files. Sure, this would not only include downloads but the more information the better ;)

Example:
inotifywait -r -m -e create -e modify $(cat /var/lib/docker/image/aufs/layerdb/mounts/$(docker inspect --format {{.Id}} adf1b94177ec)/mount-id)

I would be happy with that. It's IMHO better then not being able to catch the downloads + one can for example detect compromised binaries.

Btrfs driver should also be kept in mind.
Overlayfs suffers some inotify implementation, not sure if it would work.

Maybe just restricting that feature to aufs and btrfs would be enough for most installations.

[bang-uin]

Using docker info the directory id can be retrieved without actually knowing the storage driver:
$(cat /var/lib/docker/image/$(docker info | grep "Storage Driver" | awk '{ print $3}')/layerdb/mounts/$(docker inspect --format {{.Id}} <YOUR_CONTAINER_NAME>)/mount-id)

Maybe using /var/lib/docker/$(docker info | grep "Storage Driver" | awk '{ print $3}')/mnt is also a better solution as it seems to exist for all storage drivers.

Combined one liner:
DRV=$(docker info | grep "Storage Driver" | awk '{ print $3}') | echo /var/lib/docker/$DRV/mnt/$(cat /var/lib/docker/image/$DRV/layerdb/mounts/$(docker inspect --format {{.Id}} <YOUR_CONTAINER_NAME>)/mount-id)/rootfs

[bang-uin]

@tnich would you mind having a look at https://github.com/bang-uin/honssh/commits/feature/fs_watch and telling me what you think about the inotify approach?

[bang-uin]

The feature is now implemented for aufs, btrfs, overlay and overlay2 in this branch https://github.com/bang-uin/honssh/commits/feature/fs_watch
I had to copy the twisted inotify implementation and made more failure tolerant as it raised an exception if it failed to add an watch and killed the whole functionality. As we need to do a recursive watch failing the whole mechanism because of one failed watch is not acceptable.
Maybe i'll raise a feature request on the twisted repo, maybe ;)

Only IN_CREATE and IN_MODIFY are handled so created and modified files cannot be removed from the overlay directory.

Also only files with size > 0 are copied to handle zeroing them out.
This is just a simple check and shouldn't be the finale implementation, but for now it's working :)

Ideas are welcome!

[bang-uin]

Btw, i suggest everyone to use overlay2 as it is damn fast 😄

[unixfox]

@bang-uin You should make wiki pages for document your commits, how to use overlay2 and other things you could share 😉.

[bang-uin]

These are just hints and infos for everyone who likes to test that branch. I'll create a wiki page if this branch gets at some point merged into the master.