When monitoring or debugging your web applications, it’s often useful to know the size of the requests and responses that are being sent and received. In this article, we’ll discuss how to count incoming traffic lengths in the Play Framework using Scala.
Suppose, we have a Play Action that accepts json data:
class TrafficDemoController @Inject()(val controllerComponents: ControllerComponents)
extends BaseController with Logging {
implicit val ec: ExecutionContext = controllerComponents.executionContext
def simpleJsonAction: Action[JsValue] = Action(parse.json) { request =>
logger.info(s"Incoming body: ${ request.body }")
Ok("Ok")
}
}
Native approach: using Content-Length header
A simple way to count the size of an incoming request is to read the “Content-Length” header:
def countIncomingTrafficNaive: Action[JsValue] = Action(parse.json) { request =>
val incomingLength = request.headers.get("Content-Length").getOrElse(0)
logger.info(s"Incoming body ($incomingLength): ${ request.body }")
Ok("Ok")
}
Let’s test it:
curl -X POST -H "Content-Type: application/json" -d "{ \"a\": \"b\" }" http://localhost:9000/incomingNaive
App logs show the following:
2023-08-07 21:10:58 INFO controllers.TrafficDemoController Incoming body (12): {"a":"b"}
However, while this approach is straightforward, it won’t work for chunked requests, where the “Content-Length” header is not present.
Let’s try to test in with chunked http request. For this we’ll use the following simple Python script:
import requests
# Break data into chunks of size `chunk_size`
def chunked_data(data, chunk_size=10):
for i in range(0, len(data), chunk_size):
yield data[i:i+chunk_size]
url = "http://localhost:9000/incomingNaive"
json_data = """{ "a": "b", "c": "d" }"""
# Send the request
headers = {'Transfer-Encoding': 'chunked','Content-Type': 'application/json'}
requests.post(url, data=chunked_data(json_data), headers=headers)
This time the application logs show the following:
2023-08-07 21:21:00 INFO controllers.TrafficDemoController Incoming body (0): {"a":"b","c":"d"}
Advanced Approach: handling chunked requests
To properly handle chunked HTTP requests, we must tally the length of each individual chunk and then aggregate these lengths to obtain the total size. For this we can create a custom BodyParser that incrementally calculates the length as each chunk is received, without the need to store the entire request in memory.
Here is an example of BodyParser that counts body size:
import play.api.libs.streams.Accumulator
import play.api.mvc.{Action, BaseController, BodyParser, ControllerComponents}
/// ...
implicit val ec: ExecutionContext = controllerComponents.executionContext
val countingBodyParser: BodyParser[Int] = BodyParser("countingBodyParser") { _ =>
Accumulator {
Sink
.fold(0) { (total, bytes: ByteString) =>
val chunkSize = bytes.length
val newTotal = total + chunkSize
logger.info(s"Received chunk of length: $chunkSize bytes, total so far: $newTotal bytes")
newTotal
}
.mapMaterializedValue { _.map(Right(_)) }
}
}
It creates a custom Play Accumulator by providing it Akka Streams Sink, that accumulates the total chunks size.
Checking how it works now:
def countIncomingTrafficBetter: Action[Int] = Action(countingBodyParser) { request =>
val incomingLength = request.body
logger.info(s"Incoming length: $incomingLength bytes")
Ok("Ok")
}
It seems to work properly, now the logs show the following:
2023-08-07 21:54:49 INFO controllers.TrafficDemoController Received chunk of length: 10 bytes, total so far: 10 bytes
2023-08-07 21:54:49 INFO controllers.TrafficDemoController Received chunk of length: 10 bytes, total so far: 20 bytes
2023-08-07 21:54:49 INFO controllers.TrafficDemoController Received chunk of length: 2 bytes, total so far: 22 bytes
2023-08-07 21:54:49 INFO controllers.TrafficDemoController Incoming length: 22 bytes
But of course, there’s not much use of such parser, because it completely ignores the incoming body, leaving us with only body size it hands. Fortunately, akka-streams library provides a way to combine multiple Sinks into one, so we can come with a custom parser that is a result of combining two another parsers:
def combinedParser[A, B](aParser: BodyParser[A], bParser: BodyParser[B]): BodyParser[(A, B)] =
BodyParser(s"CombinedParser: $aParser + $bParser") { request =>
val sinkA = aParser(request).toSink
val sinkB = bParser(request).toSink
val sinkCombined = Flow[ByteString]
.alsoToMat(sinkA)(Keep.right)
.toMat(sinkB)(Keep.both)
.mapMaterializedValue { case (aFuture, bFuture) =>
// combine two Future[Either[_, _]] values into one:
for {
aEither <- aFuture
bEither <- bFuture
} yield for {
a <- aEither
b <- bEither
} yield (a, b)
}
Accumulator(sinkCombined)
}
We can make use of this parser as following:
def countIncomingTrafficFinal: Action[(Int, JsValue)] =
Action(combinedParser(countingBodyParser, parse.json)) { request =>
val (incomingLength, jsonBody) = request.body
logger.info(s"Incoming body($incomingLength): $jsonBody")
Ok("Ok")
}
Trying to call it:
2023-08-07 22:13:49 INFO controllers.TrafficDemoController Received chunk of length: 10 bytes, total so far: 10 bytes
2023-08-07 22:13:49 INFO controllers.TrafficDemoController Received chunk of length: 10 bytes, total so far: 20 bytes
2023-08-07 22:13:49 INFO controllers.TrafficDemoController Received chunk of length: 2 bytes, total so far: 22 bytes
2023-08-07 22:13:49 INFO controllers.TrafficDemoController Incoming body(22): {"a":"b","c":"d"}
As we can see, now both parsers are working and the app printed both request body length and json content.
You can find the complete code for this tutorial in our GitHub repository.
Conclusion
This technique provides a flexible and powerful way to handle both standard and chunked requests, offering valuable insights into the behavior and performance of your Play Framework applications. Whether you’re building large-scale systems or simply keen to understand the data flow in your application, this method can be a key part of your toolkit.