* Docker caching for devel
Use more caching features of Docker for building the devel image.
Downsite of this is that for users who heavily build the image it will
eat up more disk space then before.
But I think think only developers are affected here and all others will
pull from the registry.
A regular docker cleanup on the machines will solve this problem.
* Upgrade python to python3
The main Dockerfile will build now a python3 based image.
A copy of the old python2 file is still available for further testing.
* setup.py work in progress. should install library now.
* add incremental for versioning
* Fix 'ls -d' switches (worked wrong way around) and capitalisation in error message.
The data_path has changed to etc/. I'm not really happy with this but
I didn't had a better idea since we allow configs to be in ., etc/ and
/etc. Maybe we wanna changes this behaviour when we have a stable docker
release.
If the userdb.txt is not found Cowrie will load a default list.
The parser is now also a bit less error prone when parsing this file.
Some developers have different file permissions in their source
directory. This could lead to unusable Docker images.
We enforce now the correct file permissions during our Docker build.
* Docker devel image
TL;DR Providing an docker image for local development.
I wanted to have a container which has all the needed tools installed
while developming (eg flake8, pytest, pydev, etc).
The intermediate container `devel` can now be used by PyCharm as a
Remote Interpreter and for debugging. No need to setup any local test
environments because we can now use a pre-release image.
Build the container with `docker build --target devel -t cowrie:devel
.` from within the project root directory.
While building the container I encountered a bug with the
`python:2-alpine3.8` image and (at least, could be other OS also beeing
affected) the macOS kernel trying to use `socket.SO_REUSEPORT`.
After some testing I found out that the problem is just this image. So I
could have just gone and downgraded to `python:2-alpine3.7` or switched over to
`alpine:latest`. But none of them really convinced me after some
research so I decided to switch the Docker image to
`debian:stable-slim`. The resulting image is now slightly bigger then
our previous image but should give a better experience while
debugging stuff. Bonus point is that we have a functional installation
description for Debian based systems.
* New path for twisteds dropin.cache
* Fix travis flake8 path
flake8 was checking the wrong path inside travis.
Problem here is if the path is not existing flake8 will return 0 and
makes the test pass.
Due to this fakt we have now some flake8 errors which I will handle now.
* Fix flake8 E128
* Full Docker support for output plugins
cowrie makes more fun if there are also all the output plugins available
inside the docker image. And now they are!
* Fixing wrong requirements: snappy vs python-snappy
snappy: “SnapPy is a package for studying the topology and geometry of 3-manifolds, with a focus on hyperbolic structures. It is based on the SnapPea kernel written by Jeff Weeks.”
python-snappy: “Python bindings for the snappy compression library from Google.”
We want python-snappy. ;)
* MAINTAINER argument deprecated in Docker
The MAINTAINER keyword is deprecated in Docker and is replaced with the
new syntax.
Also it is now attached to the right container and not to one we
actually throw away.
* Adding /etc/cowrie/cowrie.cfg to possible configurations
We want also to look for /etc/cowrie/cowrie.cfg as a possible
configuration.
* Write ssh host keys into /var/lib/cowrie
Dynamical data should be written into /var/lib/cowrie.
I know that OpenSSH is doing this but we are not OpenSSH and we should
have only stuff written below /var.
So /var/log/cowrie and /var/lib/cowrie.
* Moving log and dl below var/, cleanup old folders
This helps keeping the writes of the daemon in one place and makes it
easier later when building a cowrie package.
Old paths have been removed from the repository to keep it clean.
* Fixing wrong log path
Path should be var/log/cowrie not var/log
* Fixing json output
The json output was not configured to use the default log path. This has
been fixed now.
* Change order of config reads
According to @micheloosterhof there is an order of precedence here.
* Adding flake8-import-order to linters
We need some kind of linter to enforce the code style.
Doing this by hand is error prone and no one will really watch for it.
I decieded to use the default style since it's from my point of view the
best to read and more strictes version.
* Enforcing imports order on bin/
* Enforcing imports order on src/twisted/
* Enforcing imports order on setup.py
* Enforcing imports order on src/cowrie/commands
* Enforcing imports order on src/cowrie/core
* Enforcing imports order on src/cowrie/dblog
* Enforcing imports order on src/cowrie/insults
* Enforcing imports order on src/cowrie/output
* Enforcing imports order on src/cowrie/proxy
* Enforcing imports order on src/cowrie/python
* Enforcing imports order on src/cowrie/shell
* Enforcing imports order on src/cowrie/ssh
* Enforcing imports order on src/cowrie/telnet
* Enforcing imports order on src/cowrie/test
* Reformat file, improved readability
* flake8 E5,E701 src/cowrie/commands/base.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/commands/base.py
src/cowrie/commands/base.py:47:121: E501 line too long (180 > 120 characters)
src/cowrie/commands/base.py:48:121: E501 line too long (182 > 120 characters)
src/cowrie/commands/base.py:49:121: E501 line too long (149 > 120 characters)
src/cowrie/commands/base.py:50:121: E501 line too long (175 > 120 characters)
src/cowrie/commands/base.py:52:121: E501 line too long (127 > 120 characters)
src/cowrie/commands/base.py:54:121: E501 line too long (185 > 120 characters)
src/cowrie/commands/base.py:56:121: E501 line too long (130 > 120 characters)
src/cowrie/commands/base.py:57:121: E501 line too long (122 > 120 characters)
src/cowrie/commands/base.py:59:121: E501 line too long (188 > 120 characters)
src/cowrie/commands/base.py:60:121: E501 line too long (188 > 120 characters)
src/cowrie/commands/base.py:61:121: E501 line too long (145 > 120 characters)
src/cowrie/commands/base.py:63:121: E501 line too long (142 > 120 characters)
src/cowrie/commands/base.py:64:121: E501 line too long (152 > 120 characters)
src/cowrie/commands/base.py:66:121: E501 line too long (128 > 120 characters)
src/cowrie/commands/base.py:67:121: E501 line too long (123 > 120 characters)
src/cowrie/commands/base.py:72:121: E501 line too long (130 > 120 characters)
src/cowrie/commands/base.py:74:121: E501 line too long (125 > 120 characters)
src/cowrie/commands/base.py:75:121: E501 line too long (140 > 120 characters)
src/cowrie/commands/base.py:76:121: E501 line too long (133 > 120 characters)
src/cowrie/commands/base.py:78:121: E501 line too long (124 > 120 characters)
src/cowrie/commands/base.py:79:121: E501 line too long (122 > 120 characters)
src/cowrie/commands/base.py:80:121: E501 line too long (129 > 120 characters)
src/cowrie/commands/base.py:81:121: E501 line too long (150 > 120 characters)
src/cowrie/commands/base.py:83:121: E501 line too long (129 > 120 characters)
src/cowrie/commands/base.py:544:68: E502 the backslash is redundant between brackets
src/cowrie/commands/base.py:553:68: E502 the backslash is redundant between brackets
src/cowrie/commands/base.py:582:66: E502 the backslash is redundant between brackets
src/cowrie/commands/base.py:696:78: E502 the backslash is redundant between brackets
24 E501 line too long (180 > 120 characters)
4 E502 the backslash is redundant between brackets
28
* Reformat code, improved readability
* flake8 E5, E701 src/cowrie/commands/curl.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/commands/curl.py
src/cowrie/commands/curl.py:413:106: E502 the backslash is redundant between brackets
1 E502 the backslash is redundant between brackets
1
* Reformat code, improved readability
* Reformat code, improved readability
* flake8 E5, E701 src/cowrie/commands/fs.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/commands/fs.py
src/cowrie/commands/fs.py:438:74: E502 the backslash is redundant between brackets
src/cowrie/commands/fs.py:465:67: E502 the backslash is redundant between brackets
2 E502 the backslash is redundant between brackets
2
* flake8 E5, E701 src/cowrie/commands/gcc.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/commands/gcc.py
src/cowrie/commands/gcc.py:167:121: E501 line too long (635 > 120 characters)
1 E501 line too long (635 > 120 characters)
1
* flake8 E5, E701 src/cowrie/commands/ifconfig.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/commands/ifconfig.py
src/cowrie/commands/ifconfig.py:11:121: E501 line too long (145 > 120 characters)
src/cowrie/commands/ifconfig.py:13:121: E501 line too long (138 > 120 characters)
2 E501 line too long (145 > 120 characters)
2
* Reformat code, improved readability
* Reformat code, improved readability
* flake8 E5, E701 src/cowrie/commands/netstat.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/commands/netstat.py
src/cowrie/commands/netstat.py:21:121: E501 line too long (126 > 120 characters)
1 E501 line too long (126 > 120 characters)
1
* flake8 E5, E701 src/cowrie/commands/ping.py
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/commands/ping.py
src/cowrie/commands/ping.py:81:74: E502 the backslash is redundant between brackets
1 E502 the backslash is redundant between brackets
1
* Reformat code, improved readability
* flake8 E5, E701 src/cowrie/commands/ssh.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/commands/ssh.py
src/cowrie/commands/ssh.py:79:87: E502 the backslash is redundant between brackets
src/cowrie/commands/ssh.py:99:81: E502 the backslash is redundant between brackets
2 E502 the backslash is redundant between brackets
2
* flake8 E5, E701 src/cowrie/commands/sudo.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/commands/sudo.py
src/cowrie/commands/sudo.py:14:121: E501 line too long (133 > 120 characters)
src/cowrie/commands/sudo.py:15:121: E501 line too long (166 > 120 characters)
src/cowrie/commands/sudo.py:16:121: E501 line too long (122 > 120 characters)
src/cowrie/commands/sudo.py:24:121: E501 line too long (133 > 120 characters)
src/cowrie/commands/sudo.py:25:121: E501 line too long (166 > 120 characters)
src/cowrie/commands/sudo.py:26:121: E501 line too long (122 > 120 characters)
6 E501 line too long (133 > 120 characters)
6
* Reformat code, improved readability
* Reformat code, improved readability
* flake8 E5, E701 src/cowrie/commands/wget.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/commands/wget.py
src/cowrie/commands/wget.py:50:19: E701 multiple statements on one line (colon)
src/cowrie/commands/wget.py:264:69: E502 the backslash is redundant between brackets
1 E502 the backslash is redundant between brackets
1 E701 multiple statements on one line (colon)
2
* flake8 E5, E701 src/cowrie/commands/yum.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/commands/yum.py
src/cowrie/commands/yum.py:8:1: F401 'datetime.datetime' imported but unused
src/cowrie/commands/yum.py:223:121: E501 line too long (193 > 120 characters)
src/cowrie/commands/yum.py:228:121: E501 line too long (193 > 120 characters)
src/cowrie/commands/yum.py:237:121: E501 line too long (193 > 120 characters)
3 E501 line too long (193 > 120 characters)
1 F401 'datetime.datetime' imported but unused
4
* flake8 E5, E701 src/cowrie/core/cef.py
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/core
src/cowrie/core/cef.py:53:121: E501 line too long (122 > 120 characters)
1 E501 line too long (122 > 120 characters)
1
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* flake8 E5, E701 src/cowrie/output/hpfeeds.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/output/hpfeeds.py
src/cowrie/output/hpfeeds.py:61:26: E701 multiple statements on one line (colon)
src/cowrie/output/hpfeeds.py:80:29: E701 multiple statements on one line (colon)
src/cowrie/output/hpfeeds.py:147:22: E701 multiple statements on one line (colon)
src/cowrie/output/hpfeeds.py:166:22: E701 multiple statements on one line (colon)
src/cowrie/output/hpfeeds.py:173:26: E701 multiple statements on one line (colon)
src/cowrie/output/hpfeeds.py:180:30: E701 multiple statements on one line (colon)
src/cowrie/output/hpfeeds.py:186:34: E701 multiple statements on one line (colon)
7 E701 multiple statements on one line (colon)
7
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* flake8 E5, E701 src/cowrie/output/sqlite.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/output/sqlite.py
src/cowrie/output/sqlite.py:73:62: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:74:60: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:82:62: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:83:60: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:91:41: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:92:67: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py💯41: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:101:67: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:109:62: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:116:45: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:117:75: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:125:45: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:126:75: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:134:41: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:135:65: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:143:67: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:149:69: E502 the backslash is redundant between brackets
src/cowrie/output/sqlite.py:165:57: E502 the backslash is redundant between brackets
18 E502 the backslash is redundant between brackets
18
* flake8 E5, E701 src/cowrie/output/mysql.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/output/mysql.py
src/cowrie/output/mysql.py:108:77: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:109:71: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:115:75: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:116:73: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:122:75: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:127:54: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:128:80: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:133:54: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:134:80: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:139:58: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:140:88: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:147:58: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:148:88: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:154:58: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:155:88: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:161:54: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:162:78: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:169:68: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:175:70: E502 the backslash is redundant between brackets
src/cowrie/output/mysql.py:191:73: E502 the backslash is redundant between brackets
20 E502 the backslash is redundant between brackets
20
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* flake8 E5, E701 src/cowrie/shell/shlex.py
Before
flake8 --count --select=E1,E2,E3,E5,E701,E901,E999,F401,F821,F822,F823 --max-line-length=120 --statistics src/cowrie/shell
src/cowrie/shell/protocol.py:37:121: E501 line too long (144 > 120 characters)
src/cowrie/shell/shlex.py:111:51: E502 the backslash is redundant between brackets
1 E501 line too long (144 > 120 characters)
1 E502 the backslash is redundant between brackets
2
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Reformat code, improved readability
* Update linters to check for E5 and E701
* Update travis file to the current state
- Removed every checker we do not use. That means twistedchecker
and pytest.
- Updated the flake8 command to match our current clean state
- Removed unused old code
* Fix format typo of cowrie.direct-tcpip.data log event
* Update travis to match the current state
Travis checks now every fixed flake8 error.
Also I removed unused code from it to keep it clean.
* Removed vim file markers
We don't want to have editor specific code in our source files.
Fix your IDE but don't do it in the code.
* src/cowrie/test/test_echo.py removed trailing whitespace
* src/cowrie/commands/yum.py removed trailing whitespace
* src/cowrie/commands/netstat.py removed trailing whitespace
* Checking for flake8 W291, W293
* flake8 F811 ./src/cowrie/commands/scp.py
I made sure to delete only the oldest version of the code.
Before
flake8 --count --select=F811 --application-import-names cowrie --max-line-length=120 --statistics .
./src/cowrie/commands/scp.py:209:5: F811 redefinition of unused 'handle_CTRL_D' from line 192
1 F811 redefinition of unused 'handle_CTRL_D' from line 192
1
* Update flake8 checks
* Fix flake8 E722, F841 src/cowrie/commands/wget.py
* Fix flake8 E722 src/cowrie/commands/ping.py
* Fix flake8 E722,F841 src/cowrie/commands/nc.py
* Fix flake8 E722, E741, F841 src/cowrie/commands/base.py
* Flake8 E731 src/cowrie/commands/base.py
I removed all lambda functions and replaced them with proper code.
That means straight function calls or not calling anything because it's
not needed.
* Fix flake8 E722, F841 src/cowrie/commands/scp.py
* Fix flake8 E722, F841 src/cowrie/commands/ssh.py
* Fix flake8 E712 src/cowrie/commands/iptables.py
* Fix flake8 E741 src/cowrie/commands/adduser.py
* Fix flake8 F841 src/cowrie/commands/tftp.py
* Fix flake8 E722 in all files
* Fix flake8 E711 bin/createdynamicprocess.py
* Fix flake8 E712, E713 src/cowrie/core/dblog.py
* Fix flake8 E712, E713 src/cowrie/shell/avatar.py
* Fix flake8 E712 src/cowrie/shell/fs.py
* Fix flake8 E712, E741 src/cowrie/commands/ls.py
* Fix flake8 E712, E713 src/cowrie/dblog/xmpp.py
* Fix flake8 E731, F841 src/cowrie/commands/gcc.py
* Fix flake8 F841 src/cowrie/insults/insults.py
* Fix flake8 F841 src/cowrie/telnet/session.py
* Fix flake8 F841 src/cowrie/commands/fs.py
* Fix flake8 E741,F841 src/cowrie/commands/last.py
* Fix flake8 E741 src/cowrie/commands/netstat.py
* Fix flake8 F841 src/cowrie/commands/free.py
* Fix flake8 F841 src/cowrie/commands/sudo.py
* Fix flake8 F841 src/cowrie/commands/curl.py
* Fix flake8 F841 src/cowrie/commands/base64.py
* Fix flake8 F841 src/cowrie/commands/service.py
* Fix flake8 F841 src/cowrie/ssh/factory.py
* Fix flake8 F841 src/cowrie/ssh/transport.py
* Fix flake8 F841 src/cowrie/output/csirtg.py
* Fix flake8 F841 src/cowrie/output/kafka.py
* Fix flake8 F841 src/cowrie/output/mongodb.py
* Fix flake8 F741 src/cowrie/output/hpfeeds.py
* Fix flake8 F741 src/cowrie/output/hpfeeds.py
* Fix flake8 F841 src/proxy/session.py
* Fix flake8 F841 src/cowrie/core/utils.py
* Update travis and Dockerfile for flake8 checks
Imports are now pep8 compatible.
Unfortunally there is no check in flake8. So this is a generall work
which has to be done manually.
Imports are now splitted into built-ins, third-party and own code.
Within this groups they are ordered alphabetical and grouped by `import`
and `from` import syntax.
Empty or not usefull docstrings has been removed from the code.
We don't need nonsense documentation eating up space.
