Writing a small app to access items in S3

 Intro

For one of our projects, we decided to use AWS S3 bucket as a storage for the recently submitted data payloads so that it would be easy to access and compare the items for a given date. The idea is that they would be stored there for three month in case someone would need to compare the payloads for the same item for different time periods and see what exactly got submitted and how it could be traced back on different stages of the item lifetime.

After doing that, it became easier to troubleshoot mapping or business logic related issues and be sure that any problem could be easily replicated if needed. For me, it is easy to access this data via cloudberry, an awesome cloud file management tool, free tier of which allows one to do anything needed for this task.

Having said that, you would still need to expose certain level of permissions to allow the user to connect to the bucket and make them locate the needed objects via some routine navigation process. This resulted in our support team asking me directly to get the files for them when they were needed and, honestly, I don't think it is worthwhile to do that all the time, so I decided to build a small Accessing tool to navigate to the needed file and download it from the browser. Thus, the support team would be able to download anything they need themselves with no pain.

Now it is time to build the tool!

let's use .Net Core with Angular2 for it. It doesn't really matter what to use here so I just want to practice a bit more with these technologies. I will omit some basic set up steps and concentrate on things I learned while I was building the tool.

AWS Permissions and Credentials management.

Today I would like to share me experience with setting up credentials access for S3Client.

There are two options here that I see right now: use roles or use users. For dev environment I think setting up a user is the most straightforward choice since anyone could pick up the code from any machine and getting access to the user details would cater for that. 

For classical .net, there was a bit in the app.config/web.config that you could add for aws sdk to read from and that would wire things up automatically. With .net core, however, there is a json file to work with, this makes things a little bit different. I found this article to be quite helpful in getting things right. The best part is that the settings are based on the environment, similarly to .net and if the system cannot find a certain environment settings file, it will default to the role based management on the EC2 instance. Neat.
The whole thing boils down to the following:

• Install the AWSSDK.Extensions.NETCore.Setup nuget package
• Add the credentials as a part of the appsettings.Development.json file
• Since I am using Dependency Injection here, I can leverage off of the core DI system - just modify Startup class -> ConfigureServices method by modifying the serice collection object like so:
  
  • services.AddDefaultAWSOptions(Configuration.GetAWSOptions());
  • services.AddAWSService<IAmazonS3>();
• Once this is done, the S3 client can be injected anywhere in the code with the credentials available to it.

There is a trick to that. Previously, you could have provided access key and secret key in the config file and it would have been picked up. This is no longer allowed, at least at the moment, since it encourages bad practices of commiting sensitive information to a source control system. I think this is a bit too much - it is possible to add variables in the config and later replace it in the deployment step. But oh well, it is what it is. 

There are several options to get passed that:

• Specify a ProfilesLocation and store credentials there. Once a location to the file is set in the config file, you can just add credentials there in a format similar to this:
  
  

  [default]
  aws_access_key_id = <access-key>
  aws_secret_access_key = <secret-key>
  
  [development]
  aws_access_key_id = <access-key>
  aws_secret_access_key = <secret-key>
  

• Run the app with appropriate environment variables or
• Programmatically add variables in runtime after reading them from the config. This would allow to bypass the restriction for the config and include credetials there. Just remember to remove/replace any sensitive data when the code is committed.

I decided to go with the last option like so:

Environment.SetEnvironmentVariable("AWS_ACCESS_KEY_ID", Configuration["AWS:AccessKey"]) Environment.SetEnvironmentVariable("AWS_SECRET_ACCESS_KEY", Configuration["AWS:SecretKey"]) Environment.SetEnvironmentVariable("AWS_REGION", Configuration["AWS:Region"])

You can read in more details about that in this thread.

Side note: If you try to use DI to call aws service from code while there is a configuration issue, the errors you might see would be as clear as mud - for instance, if credentials could not be located in an extermal profile, a runtime error would be swollen and you could only see a 500 response code from an api. To get something more meaningful, try to call the service from the startup class where it would be created from AWSConfigurationOption. Errors there are much more helpful.