ENSE 472 - Digital Networks - Laboratory
University of Regina - Engineering and Applied Science - Software Systems Engineering
Lab Instructor: Adam Tilson
The purpose of this lab is to:
A computer running Windows, MacOS or Linux, with an Intel or AMD-based processor (x86 or x86-64) with administrator privileges
OR (Recommended):
zorin, and password is zorinPreviously, we looked at low-level socket programming in Python. We observed this was very powerful, but also tedious - you needed to be aware of low-level concerns like creating threads, handling shared memory, and properly setting up sockets. Alternatively, one can use higher-level APIs to manage much of this for you. One example of this is using Twisted Python. We will use this to quickly prototype some simple applications. We will also use WireShark in this lab to examine the packets which these applications generate to get more idea of what is happening on our network.
Last week we saw sockets in python, a very low level interface for programming network applications. While this helps you to understand how the operating system handles these tasks, there were some significant challenges!
The Twisted Library in python aims to solve some of these problems. Why Twisted?
deferred system they popularized would go on to inspire similar methods, such as promises, which would be used jQuery, dojo, Node.js and JavaScripttwisted.trialwrite, writeSequence, loseConnection, getPeer, getHostmakeConnection, connectionMade, dataRecieved, connectionLostIn the next section we will look at several examples. These are based off the Twisted documentation examples, with some modifications made by me for consistency. These are Copyright Twisted Matrix, and licensed MIT.
Let’s create an echo server, as we did last lab. This is a server which simply sends back any data sent to it:
echo_tcp.py
from twisted.internet import protocol, reactor, endpoints
class Echo(protocol.Protocol): # The brackets means we are extending their class
def dataReceived(self, data): # This callback will run when we recieve data
print(f"received {data!r}") #print it out
self.transport.write(data) # Write that data back to our transport
class EchoFactory(protocol.Factory): # Rather than use a constructor, we will use a Factory
def buildProtocol(self, addr):
return Echo()
# Creates a server endpoint. Starts the EchoFactory, listening to new tcp connections on port 1234
reactor.listenTCP(1234, EchoFactory())
reactor.run()
Note: These examples use a slightly older syntax, but I find it easier to follow than the modern syntax, and it still works in modern versions. Further, many examples you will find online, inlcuding the cited textbook, use this older syntax.
Run your server with
python3 echo_tcp.py
This is server code, we will need a client to test it.
telnet localhost 1234
hint: Exit telnet with Ctrl+], then q<enter>
Note that when we were dealing with sockets, it was a significant challenge to make the server work for multiple clients at once. Try multiple clients with this server and see how it goes!
TCP is a secure communications protocol, which means that a sophisticated set of handshakes will be performed to establish connection between the client and the server. After that messages will be sent and delivered reliably to and from the server and the client. Finally messages will be sent to close the connection to the server. In the next sections we will look at these protocols in more detail, and see these packets being exchanged using WireShark.
Of course, maybe we don’t want to connect from a server, and would rather connect from a client. Let’s look at some code which can do that, and examine some of the callbacks within:
echo_tcp_client.py
from twisted.internet import reactor, protocol
class EchoClient(protocol.Protocol):
def connectionMade(self):
self.transport.write(bytes("Hello, world!", 'utf-8'))
def dataReceived(self, data):
print ("Server said:", data)
self.transport.loseConnection()
class EchoFactory(protocol.ClientFactory):
def buildProtocol(self, addr):
return EchoClient()
def clientConnectionFailed(self, connector, reason):
print ("Connection failed.")
reactor.stop()
def clientConnectionLost(self, connector, reason):
print ("Connection lost.")
reactor.stop()
reactor.connectTCP("localhost", 1234, EchoFactory())
reactor.run()
This one is a bit more involved - in our EchoFactory we have a few callbacks that are overloaded to handle errors such as connections failed and connections lost. Our EchoClient is using a simple protocol to read and write from our transport. After receiving data it is cleanly closing the connection using the loseConnection function.
Note that we did not need to make threads to handle sending and recieving data. This is due to the event-driven nature of the application, which uses a single thread to schedule tasks in a manner that stays responsive. McKellar provides the following figure in her book Twisted Network Prorgramming Essentials, 2nd, which explains this model:
In Twisted, it is relatively easy to change the protocol. Here is an example of an equivalent server using UDP:
echo_udp.py
from twisted.internet import reactor, protocol, endpoints
# We'll use the Datagram Protocol now
class Echo(protocol.DatagramProtocol):
# datagram Protocols need to identify who we are!
def datagramReceived(self, data, addr):
print(f"received {data!r} from {addr}")
self.transport.write(data, addr)
# UDP does not use streams, instead we register callbacks to handle as they come in
reactor.listenUDP(1234, Echo())
reactor.run()
Again, you may run this with
python3 echo_udp.py
UDP cannot be connected through Telnet, but on linux, you can use Netcat utility
nc -u localhost 1234
This is a connectionless protocol that is optimized for speed rather than reliability, and will thus not send all of the extra control packets. However, if a packet is missed, there is no way to know based on the protocol, so the application must handle this.
Note: The UDP echo client would be much more streamlined. For some examples of what this would be like using similar syntax to what we have used, you can check out Twisted UDP 12.3.0 or the Twisted 12.3.0 Examples
So far we have had unencrypted traffic to our server. Let’s set up the same server using SSL.
We are going to need a private key as well as a self-signed certificate:
openssl req -newkey rsa:2048 -keyout domain.key -x509 -days 365 -out domain.crt
Enter a passphrase for the PEM. Don’t forget it - you will need it every time you spin up the server when you use the key! You are going to be self-signing this certificate, so you may fill out the fields as follows.
Now we need to create our server:
echo_ssl.py
from twisted.internet import protocol, reactor, endpoints, ssl
class Echo(protocol.Protocol): # The brackets means we are extending their class
def dataReceived(self, data): # This callback will run when we recieve data
print(f"received {data!r}") #print it out!
self.transport.write(data) # Write that data back to our transport!
class EchoFactory(protocol.Factory): # Rather than use a constructor, we will use a Factory
def buildProtocol(self, addr):
return Echo()
context = ssl.DefaultOpenSSLContextFactory("domain.key", "domain.crt")
reactor.listenSSL(1234, EchoFactory(), context)
reactor.run()
Connecting to an SSL server is beyond the powers of a Telnet client. Let’s try the OpenSSL client instead:
openssl s_client -connect localhost:1234
You will find this is a very verbose application, displaying all of the data that was exchanged, including the certificate (which contains the public key) and the session ticket. However, from an end user perspective we will not be able to tell any difference!
This is an end-to-end encrypted protocol. We will get to examine this protocol in more depth in section 3. Please note that in 2023 this protocol is deprecated in favour of TLS for performance and security concerns, however we will still be able to use it to investigate encryption.
Note: Modern versions of Twisted use a much more complicated TLS setup than our simple SSL. It might be worth checking out this version of the documentation for help with this: Twisted Documentation: Using SSL in Twisted
Recall that our model of a network has various physical configurations:
However, this is only half of the story. There is a complex sequence of layers and packets which are required to make this physical layer work with our applications:
The data for our application exists in the upper-most layer (application). Everything below this is for transmission protocols and addressing.
There is quite a few fields inside each of these packets. Here is a simplified view with some of the contents which are of relevance to us.
For a more complete overview of the packet contents, one may refer to the packet standard.
This is an example of a TCP data packet. There are also TCP control packets, and other types of packets that don’t use TCP, but instead use some other protocol.
Additionally, we can look into the IP Packet Format:
At the hardware level (ethernet), the only thing being sent are strings of bits, (1’s and 0’s). However, if we know these protocols, we could put back together the contents of the information being sent.
One way to do this would be to use some kind of hardware monitor, which would physically exist on the network, however these are very costly.
In the next section we will use WireShark to capture some of these packets to see what’s happening inside.
WireShark is a free and open-source packet analyzer. It has multiple uses, such as monitoring network traffic, network troubleshooting, and protocol development.
You can download wireshark from their downloads page:
On linux, you will need to run this as a superuser, (sudo wireshark) and as an Admin on Windows. If you are in the lab I can help you get set up with this.
Once you pop into WireShark, you can see a list of the interfaces. We are going to get started using the loopback (localhost) interface
Double-click this to begin capturing packets on the loopback address.
Let’s try printing a simple dialogue using the TCP echo app:
As soon as you connect over TCP, you should see some packets be exchanged, even before a message is sent. These are control packets establishing a connection
Send a hello message and observe that additional packets appear:
Finally, we can close our connection on either side, and additional packets will be exchanged.
Also, note if you let this run for a while, you will start to see a multitude of DNS packets. These are routine lookups. If you try this on an actual network adapter, you will see a flood of other packets, which will require filtering to work.
Let’s take a look inside and see what some of these packets are. To begin, click on the packet which has a [PSH, ACK] flag set beside it.
In the lower left, expand the Internet Protocol v4 section, the Transmission Control Protocol Section, and the Data section. By looking through here we can see lots of useful information:
As we know, there should be a corresponding message sent back in the other direction. See if you can find it. How will you know you have found the correct one?
Let’s look at an example of a chat application running in Twisted Python:
tcp_server.py
from twisted.internet import reactor, protocol, endpoints
class Server(protocol.Protocol):
def __init__ (self, users):
self.users = users
self.name = None
def connectionMade(self):
print ("New Connection")
self.users.append(self)
print ("A new connection has been made. Geting name...")
self.transport.write("Welcome to the Server, what is your name?".encode("utf-8"))
def dataReceived (self, data):
message = data.decode("utf-8")
if self.name:
if message == "!Q":
print (f"<{self.name}> left the chat.")
for user in self.users:
if user != self:
user.transport.write(f"<{self.name}> has left the chat!".encode("utf-8"))
self.users.remove(self)
else:
print (f"<{self.name}>: " + message)
for user in self.users:
if user != self:
user.transport.write(f"<{self.name}>: ".encode("utf-8") + data)
else:
if message == "!Q":
print ("Unnamed user left")
else:
self.name = data.decode()
print (f"A new connection is now known as <{self.name}>")
for user in self.users:
if user != self:
user.transport.write(f"<{self.name}> has entered the chat".encode("utf-8"))
else:
user.transport.write(f"Welcome to the chat <{self.name}>. Please type !Q to quit.".encode("utf-8"))
class ServerFactory (protocol.ServerFactory):
def __init__(self):
self.users = []
def buildProtocol(self, addr):
return Server(self.users)
if __name__ == "__main__":
reactor.listenTCP(1234, ServerFactory())
reactor.run()
tcp_client.py
from twisted.internet import reactor, protocol, endpoints
class Client(protocol.Protocol):
def __init__(self):
reactor.callInThread(self.send_data)
def dataReceived(self, data):
data = data.decode("utf-8")
print (data)
def send_data(self):
while True:
message = input()
self.transport.write(message.encode("utf-8"))
if message == "!Q":
reactor.stop()
break
class ClientFactory(protocol.ClientFactory):
def buildProtocol(self, addr):
return Client()
if __name__ == "__main__":
reactor.connectTCP("localhost", 1234, ClientFactory())
reactor.run()
This is a server-client application: users connect to the server, and then issue chat commands which are sent to all other users. Using what you know about Wireshark, are you able to eavesdrop on conversations that are sent over the network?
As with the previous assignment, this one will be completed in three phases. Please take a checkpoint after each phase of your code.
Phase 1: Using the Twisted Chat application we developed in Part 4 and WireShark, create a session with one server and two clients. Send a messages from the first client to the second client. Using WireShark, examine the packets. In a text document, using screen captures, show some annotated captures from these packets - in particular show which order the packets travelled. Flag anything which might identify source, content, or message. Are these messages secure? Why or why not?
Phase 2: Upgrade the Twisted Chat application to use end-to-end encryption with SSL. Using the encrypted version, repeat the packet collection process from phase 1, again documented in a text file. Which Protocol is being used now? Are you able to see the messages being sent? Would an end-user be able to tell that anything different is happening that in the TCP Client? As before, flag anything you can, such as certificates being exchanged.
Phase 3: Modify the Twisted Chat application to use UDP instead of TCP. Note that UDP does not have any notion of connections, thus you will need to simply assume the first Datagram recieved from the user is their name, and subsequent sends are chat messages until the disconnect message is recieved. Repeat the packet collection and documentation excercise from phase 1. What has changed on the network side? Which benefits and drawbacks of this technology are apparent compared to TCP by examining the code and packets?
Please submit your lab documents and code for phases 2 and 3 to URCourses by the due date. If you are working with a partner, you need only submit one copy, but please include both partners names on all assignments.
J. McKellar, Architecting an event-driven networking engine: Twisted Python, Philly ETE 2013, [Online]
| Baeldung, Creating a Self-Signed Certificate With OpenSSL | Baeldung, 2022, [Online] |
Twisted Python, Home-page examples, [Online]
| A. Ivanov, Twisted Python Chat, TCP | Python, [Online] |
J. McKellar, Twisted Network Programming, 2nd Edition, 2013, O’Reilly Media. [(Available free with your SafariBooks Account)](https://learning.oreilly.com/library/view/twisted-network-programming/9781449326104/)
Twisted Docs 12.3.0 TCP UDP SSL Core Examples