Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Accouting Service (WIP) #428

Closed
wants to merge 4 commits into from
Closed

feat: Accouting Service (WIP) #428

wants to merge 4 commits into from

Conversation

fforbeck
Copy link

@fforbeck fforbeck commented Oct 8, 2024

Add Accounting Service for Egress Metrics Tracking

This PR introduces a new accounting service to track egress metrics for billing purposes. The service integrates with both DynamoDB and Stripe to ensure accurate and efficient billing for our customers.

Key Features:

  • DynamoDB Integration:

    • Tracks egress data metrics to provide a reliable source of truth for data usage.
    • Ensures data consistency and availability for billing calculations.
  • Stripe Integration:

    • Automates billing processes based on the tracked egress metrics.

Next Steps:

  • Tests to verify the accuracy of data tracking & integrations (DynamoDB and Stripe).
  • Monitor the service in production to ensure stability and performance.
  • Documentation

Tasks:

  • DynamoDB Table for Egress Events [Accounting Service]
  • SQS Queue for Egress Events [Accounting Service]
  • AWS Lambda Handler for Event Processing [Accounting Service]
  • Stripe Integration for Billing [Accounting Service]
  • Testing and Validation [Accounting Service]
  • Monitoring and Logging [Accounting Service]
  • Documentation [Accounting Service]

Copy link

seed-deploy bot commented Oct 8, 2024

View stack outputs

@seed-deploy seed-deploy bot temporarily deployed to pr428 October 9, 2024 19:24 Inactive

// simulate the SQS event that triggers the handler
// FIXME (fforbeck): why the events are not collected?
const collected = await collectQueueMessages(ctx.egressQueue)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@travis, do you have any ideas on why it would not collect the events from the queue?

}))
}

const response = await ctx.egressHandler(sqsEvent, ctx)
Copy link
Author

@fforbeck fforbeck Oct 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@travis, Do you have any suggestions on how to test the handler?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

per my comment below, we may not need this at all - happy to chat more!

assert.equal(record.ok.results[0].resourceId, e.resourceId)
assert.equal(record.ok.results[0].timestamp, e.timestamp)
}
// FIXME (fforbeck): how to check we send the events to stripe?
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@travis, How we can test if the Stripe Usage Record API is being called for each event?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

goooood question - I think the most sophisticated test logic we have for stripe is here:

https://github.com/storacha/w3infra/blob/main/upload-api/test/billing.test.js

it might be worth trying to use this:

https://github.com/stripe/stripe-mock

but I don't think we've used it for anything yet

it's also worth looking into this:

https://docs.stripe.com/billing/testing

high level though I think our Stripe testing overall can be improved! happy to kick around ideas for this!

*/
async function sendRecordUsageToStripe(stripe, egressEvent) {
const subscriptionItem = {
id: 'sub_123', // FIXME (fforbeck):
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@travis, we currently don't have this subscriptionItem information available. I was wondering if that would be the exact same ID for all the customers, or if we can find it per customer somehow.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's going to be different for each customer - I think this is the interface you're looking for - it should let you find a Stripe ID (account in that interface) given one of our customer IDs (customer in that interface) - the naming is a little confusing given that we call "customer IDs" "accounts" in other places - sorry about that!

@travis
Copy link
Contributor

travis commented Oct 10, 2024

I still owe this a close review, but high level wanted to mention that as I've been thinking about this I've come to think this should probably be sitting behind a UCAN invocation, largely because I think we want to be able to show receipts to customers that are signed by the gateway. I'm imagining a capability like usage/record with an nb field that looks like the schema you've put together here:

https://github.com/storacha/w3infra/pull/428/files#diff-39796372fa47ce3ebdac6faee1fe4120794908a828509fa05c09a4833ea6ed04R11

we already have usage capabilities and I think this slots pretty nicely in there:

https://github.com/storacha/w3up/blob/main/packages/capabilities/src/usage.js

given this we can probably just get rid of the SQS queue - I'm not sure it adds much if each of these invocations is handled by a separate lambda anyway

@fforbeck fforbeck closed this Oct 17, 2024
@fforbeck fforbeck reopened this Oct 17, 2024
@fforbeck
Copy link
Author

Closing. New PR: #430

@fforbeck fforbeck closed this Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants