Hi All,
Script below for the impatient :)
Its been two weeks now. A lot longer than a normal person would take to write a threaded application in python. But persistence pays off and when the script finally runs you feel like jumping, swearing and crying all at the same time. I hope this explanation of how to setup a multithreaded port scanner will help other people. I am going to dumb the explanation down for my own purposes. I warn this is my own interpretation of threading and probably not the best way to do it but it worked for me and may help someone else out.
THREADS, WORKERS and QUEUES....
So here goes. By default python is a single threaded application. What is a thread. As wikipedia describes. I am going to use the post office as an analogy.
Wiki Terms
In computer science, a thread of execution is the smallest sequence of programmed instructions that can be managed independently by a scheduler (typically as part of an operating system)
My Awesome Lamen Terms
Process of sending mail
To make python run faster you have to specify multiple threads to do multiple jobs.
Worker processes
Wiki terms
Couldnt find one.
My Awesome Lamen Terms
The employees.
Queues
Wiki terms
Couldnt find one.
My Awesome Lamen Terms
The pile of mail that is waiting when the workers show up for work... Late :-p
How do threads, queues and workers work together (in a story)?
Well the CEO (python script writer) hires 10 people (workers) to rock up to work. When they rock up to work they notice that there is nothing to do. So as all good CEOs do, they assign out a list of jobs in the form of mail (queues / ports to scan). Before assigning out the jobs the CEO compiles a list as big as he deems necessary. For this instance it was 100 jobs (ports). Now he has 100 jobs he asks that each worker grab the mail and start sending the mail out. One at a time they go out until all mail is sent out. All the workers can now exit and go home.
What took me so long?
Now I had a heap of these issues and I will details some of them now.
Creating too many threads
Initially when I was creating threads I didn't know how to create a for loop to cycle through 100 ports and then only create 10 threads. My initial script looked like this.
for p in range (portLength):
thread.start_new_thread(portScan, (p, destIP))
Well this created 100 threads. Bad idea. So lets create worker processes to handle the threads instead. Then the next issue comes...
Worker process hanging stating 'none'
So if you read the script below it shows you how I created the worker process.
Up until today I could not work out why my script was hanging whilst threading. I am sure there is a reason but I would need to do some more digging around.
Debugging
I used the pdb module and verbose logging as shown in the example below. This did not lead me to the issue but still a good process.
PDB supports setting (conditional) breakpoints and single stepping at
the source line level, inspection of stack frames, source code listing, and
evaluation of arbitrary Python code in the context of any stack frame. It also
supports post-mortem debugging and can be called under program control
http://pymotw.com/2/argparse/
Executed like:
python -m pdb -v test.py
This did not throw any errors to the console but it led me to the next section and probably my best tip when it comes to any form of scripting.
Script (More information below script)
#Import Modules
from scapy.all import *
from Queue import Queue
from threading import Thread
import argparse
import sys
#Set Variables
threadCount = 10
destIP = "192.168.136.131"
q = Queue(maxsize=0)
#Empty Arrays
portList = []
openPorts = []
closedPorts = []
threads = []
def main():
#Parser Arguments and Usage
parser = argparse.ArgumentParser(description='This script performs a Syn scan on a specific host with the ports that are specified by you.', usage='Specify Ports with the -p switch. Port range -p 80-200, specific port -p 80, multiple ports -p 80,443,25,22', add_help=True)
parser.add_argument('-p', action='store', dest='argPorts', help='Type ports in like follows. For a range -p 80-200, for a group of ports -p 443,80,25,22 or single port -p 80', required=True)
parser.add_argument('-d', action='store', dest='destIP', help='Type dest IP in this field. i.e. 192.168.0.5', required=True)
parser.add_argument('-t', action='store', dest='threadCount', help='Type the amount of threads to use as an integer', type=int, required=True)
if len(sys.argv)<3:
parser.print_help()
sys.exit(0)
options = parser.parse_args()
argPorts = str(options.argPorts)
destIP = str(options.destIP)
threadCount = int(options.threadCount)
#If argument has hyphen split it then create a list from the range
if '-' in argPorts:
print "argPorts is a range"
startPort = argPorts.split('-')[0]
endPort = argPorts.split('-')[1]
for p in range(int(startPort), (int(endPort) + 1)):
portList.append(p)
print portList
#If argument has comma, split then create list.
elif ',' in argPorts:
print "argPorts is a comma seperated list"
for p in argPorts.split(','):
portList.append(int(p.strip()))
else:
portList.append(int(argPorts))
for i in range(threadCount):
worker = Thread(target=qProcessor, args =(q,))
worker.setDaemon(True)
worker.start()
for p in portList:
q.put(portScan(p, destIP))
q.join()
def qProcessor(q):
while True:
q.get()
q.task_done()
def portScan(port, dstIP):
scan = sr1(IP(dst=dstIP)/TCP(dport=port,flags="S"), verbose=0)
print "Scanning port %s and IP: %s"%(port, dstIP)
if scan.getlayer(TCP).flags == 0x12:
openPorts.append("IP: %s \t Port: %s \t Status:Open"%(scan.getlayer(IP).src, scan.getlayer(TCP).sport))
Issue here # sr1(IP(dst=dstIP)/TCP(dport=port,flags="R"),verbose=0)
if scan.getlayer(TCP).flags != 0x12:
closedPorts.append("IP: %s \t Port: %s \t Status:Closed"%(scan.getlayer(IP).src, scan.getlayer(TCP).sport))
if __name__ == '__main__':
main()
for port in openPorts:
print port
THE BEST ANSWER - KEEP IT SIMPLE STUPID
When
you have issues with things hanging in your script replace the action
portions of your functions with simple things like 'Print I am %s'
%(variable). This will let me know where in your script things may be held up. In my
case I specified what port I was scanning.
So where
did the script stop. On that damn reset!!! (highlighted in red) above.
Tips for working out issues in your script.
So as I said above. Keep it simple. When you start to get into complex scripts that have multiple functions and classes it is quite easy to gloss over what is and isn't working. This is stating the obvious but here are some of my tips for understanding where my scripts are not working.
Print Print Print
When you have a script that iterates through a list when you first script it up, print out what is going on. For the above example you can see
that I am printing 'scanning port and ip'.
Enable Verbose mode and debugging
Some of the debugging information may look like gobly gook to a lot of you. Including me. But using your google fu you can look into what module is being created.
Exceptions
Try, except. This is an even better place to start over debugging. What I did to narrow it down was this.
try:
func(ip,port)
except Exception as e:
print "Error in func function"
print e
Smaller Steps
So when writing a script take baby steps. Don't try and add everything at once when you are a beginner scripter. It doesn't help you find issues quickly. So what do I mean. Well here is an example from my port scanner.
- Create simple port connect utility with one port and one IP
- Create function to call port connect, one port one IP
- Create function to call range of ports with one IP
- Thread the application so it takes a fixed amount of ports
- Add an argument parser.
You can add a heap more above if you wanted but if you see my flow I am only ever working on one issue at a time. I am not battling my function, threading and argument parsing all in one. I make smaller scripts like building blocks and add to them until I have something I like.
Hopefully this wasn't too big of a post but helpful none the less. Take it easy.
