0

Amazon Web Services SQS service allows you to define a policy that lets unauthenticated users submit simple POSTs to the queue without signing the requests.

I'd like to do the same thing, but with AWS Kinesis, so that I can collect data from devices and processes that use webhooks to post notifications, but are not able to sign the requests.

Is this possible?

4
  • Does this meet your needs? docs.aws.amazon.com/apigateway/latest/developerguide/…
    – Parsifal
    Aug 10, 2020 at 21:58
  • It does, but I'll be doing a very high volume of operations, and I'm trying to avoid the $1 per million requests when using the API gateway. Aug 11, 2020 at 2:35
  • This was just announced, looks exactly like what you want: aws.amazon.com/about-aws/whats-new/2020/08/…
    – Parsifal
    Aug 20, 2020 at 22:43
  • @Parsifal that's cool, and it will certainly make it easier to do what I want (I was already planning on using API gateway and writing a custom mapping template to Kinesis), but I was hoping to avoid the expense of the API gateway by making anonymous calls directly to Kinesis. Aug 21, 2020 at 15:40

2 Answers 2

0

I still don't think you would want to have absolutely no protection on your AWS resource. You may be able to use a resource based policy on ip but this is considered bad practice. The only reason you would have large costs with the API GW + Kinesis is if clients are sending excessive numbers of requests all containing tiny amounts of data, which is not what you want in any situation. But I imagine you would still want API GW as you can perform throttling and stop costs growing exponentially, & later DDOS attacks or simply mitigate bugs in client's code. API GW should be saving you money not costing you more.

What is more desirable is if you have clients send you data in intervals say every x minutes or y seconds. This can be a burden on the client to do however, and you may need to produce a sdk, daemon or service they can make use of which will do this for them automatically, running with their app listening on a particular port for data. A good example of this is the X-Ray daemon AWS produces which batches data transfer to the cloud. Without this strategy it is likely that the service would be exponentially more expensive. This in turn means you may have only 1% of the number of requests going to your API GW than you would have otherwise had.

One final note is that although batching many messages excessively together will in-turn lead less accurate real-time analytics. You need to keep in mind there are limits as to how quickly you can increment a given data field especially over distributed systems which are inherently eventually consistent. I learnt this lesson via pain when I was creating an aggregation table in DynamoDB. I was calling inc very quickly on an item, and this causes many Transaction errors as well as using up many more WCUs than I actually needed. So even though my table had more WCUs and my batches were tiny. It was less accurate than if I would have batched writes up into lets say 10-30 second intervals.

Response to Comment

It's not clear what you mean by events, I'll assume you do 1 HTTP request for each event. If one device is doing 1k HTTP requests a second that would be impressive! Although probably not possible! I'll take your word for it ( ;

API 1 mill req a sec

Lets say you have 1k devices, each doing 1k request per sec, which will be a total of 1 million a second, with 2.628e+6 seconds in a month, this would mean 2.63e+12 requests per month, which would cost you in excess of 2 million dollars per month going by their pricing and lowest cost of 0.90 per million. You'd probably be the biggest spender on API GW in the world! Congratulations.

API Batching savings

Unless you have deep pockets, that's a problem! Even at 100 times less the cost that will still be expensive! But lets say you send data every 60 seconds, that means 1k devices x 2.628e+6 sec in month / 60 req/sec. That would only be a bit more than $40. But that would assume you could bunch up 60k events in 1 request, might not be feasible, but it does illustrate the savings.

Batching savings in general

There is simply now way to do what you are talking about economically without batching. At many different layers of infrastructure batching is an efficient way to reduce cost and increase performance. SQS does this when you read and write more than 1 message at a time. DynamoDB costs much less when you batch writes and reads, EBS throughput optimized volumes are another great example, as it buffers writes. Even if you could hit Kinesis directly, without batching it would cost a fortune! Let's crunch the numbers:

Kinesis

Each shard supports 1k writes per second which means 1,000 x shards, which means 1.44$ x 1k per day which is well over 40k a month!

TL;DR

Anything other than changing the devices to send larger requests less frequently, will be unaffordable at scale no matter how you skin the cat.

1
  • I don’t control the devices or workflow on the reporting end, so I can’t put them into batches. I can supply a web hook, which will receive a POST whenever an event occurs. Each device will be generating 10-1000 events per second, and there are hundreds of devices, so cost becomes a significant factor with the API gateway. Aug 24, 2020 at 17:05
0

You could setup a Cognito Identity Pool and enable unauthenticated/anonymous access. Associate an IAM role with the unauthenticated/anonymous users that allows writing to Kinesis.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.