Spring Boot 3 with AWS S3 SDK
Exploring the AWS Java SDK and spring-cloud-aws to manage files in AWS S3
7 min readSep 27, 2023
Problem Statement:
Let’s assume we’re asked to implement a service which allows us to upload, store and download files.
Functional Requirements:
- Users should be able to upload, download and delete files.
- The service should support any conventional file type. ( .txt, .doc*, .mov, .mkv, .pdf etc.)
- Users should be able to see at least 20 previous versions of the file and download / see the content.
- The service should support large files. (Maximum 50GB)
- The user should be able to pause and resume the upload and download operation at a point in time.
Non-functional Requirements:
- Every request should be secured. i.e. properly authenticated and authorized. The user should have access to only their files.
- Scalability, reliability, consistency and speed of the APIs.
The requirements are what is expected of any Cloud Storage Application like Dropbox, Google or iCloud. I’ll discuss requirements 1,2 & 3 in this post and discuss 4 & 5 in Part 2.
High-Level Solution:
- We’ll implement the APIs with spring-boot 3.0.1
- For storage, we’ll be using AWS S3
- For the AWS SDK, we’ll use the spring-cloud-aws dependency.
<dependency>
<groupId>io.awspring.cloud</groupId>
<artifactId>spring-cloud-aws-starter-s3</artifactId>
<version>3.0.1</version>
</dependency>4. For authentication and authorization, we’ll be using AWS Cognito.
spring.config.import=classpath:env.properties
spring.security.oauth2.resourceserver.jwt.issuer-uri=${JWT_ISSUER_URI}
spring.cloud.aws.credentials.access-key=${AWS_ACCESS_KEY}
spring.cloud.aws.credentials.secret-key=${AWS_SECRET_KEY}Authentication and Authorization
We used AWS Cognito as the authorization server and the spring-boot application as a resource server.
- AWS Cognito: Although lacking some features it's a very easy and cheap solution for implementing OAuth2.0-based user sign-up, login and access control. One of the major drawbacks of the Cognito service is that it does not allow custom claims in the Access token. You’ll be able to add custom claims to the ID token, however, it's not so useful when you’re trying to implement granular user access control in the resource servers. For this demo, I used the user pool groups as the authorities and added users to the groups for assigning authorities. After that, the Access token looked like this:
// access token
{
"sub": "dfd14830-74d1-40ef-8115-2760aef06098",
"cognito:groups": [
"file-service:read",
"file-service:admin"
],
"iss": "https://cognito-idp.{{aws-region}}.amazonaws.com/{{user-pool-id}}",
"version": 2,
"client_id": "{{client-id}}",
"origin_jti": "a1s90s9-o1c9-p130-d10b-e691m3ob394d",
"event_id": "1b272be5-t614-u67f-p671-fq3991znjd",
"token_use": "access",
"scope": "openid profile",
"auth_time": 1694892919,
"exp": 1695753834,
"iat": 1695750234,
"jti": "0c44c5u5-f6bz-7119-9fd9-4bf9563o2a74",
"username": "dfd14830-74d1-40ef-8115-2760aef06098"
}
- Spring-security: I configured the spring-security to use the cognito: groups claim as the authorities. I also used a method-level authorization. However, the same functionality could be achieved by using a request-matcher-based authorization, it seemed cleaner to use method level authorization.
@Configuration
@EnableWebSecurity
@EnableMethodSecurity
public class SecurityConfig {
@Bean
public SecurityFilterChain securityFilterChain(HttpSecurity httpSecurity) throws Exception {
httpSecurity.authorizeHttpRequests(auth -> auth
.requestMatchers("/health/check").permitAll().anyRequest().authenticated());
httpSecurity.oauth2ResourceServer((oauth) -> oauth.jwt(Customizer.withDefaults()));
return httpSecurity.build();
}
@Bean
public JwtAuthenticationConverter jwtAuthenticationConverter() {
JwtGrantedAuthoritiesConverter jwtGrantedAuthoritiesConverter = new JwtGrantedAuthoritiesConverter();
jwtGrantedAuthoritiesConverter.setAuthoritiesClaimName("cognito:groups");
jwtGrantedAuthoritiesConverter.setAuthorityPrefix("");
JwtAuthenticationConverter jwtAuthenticationConverter = new JwtAuthenticationConverter();
jwtAuthenticationConverter.setJwtGrantedAuthoritiesConverter(jwtGrantedAuthoritiesConverter);
return jwtAuthenticationConverter;
}
}- IAM Permissions: I created a custom IAM policy and attached the policy to a user. I then used the access token in the sping-boot configuration to give it access to S3. Although, it’s recommended to create a role and assign the role temporarily to the host EC2, using the access and secret key was much simpler for this demo. The actions are restricted to only the root BUCKET demo-s3-bucket-ca-central-1.
spring.cloud.aws.credentials.access-key=${AWS_ACCESS_KEY}
spring.cloud.aws.credentials.secret-key=${AWS_SECRET_KEY}{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketVersions",
"s3:GetBucketVersioning"
],
"Resource": "arn:aws:s3:::demo-s3-bucket-ca-central-1"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:GetObjectVersion",
"s3:DeleteObjectVersion"
],
"Resource": "arn:aws:s3:::demo-s3-bucket-ca-central-1/*"
}
]
}API Implementation:
- Upload Files: I used the S3Operations API for uploading the file. The spring-cloud-aws dependency automatically injects the required beans and it abstracts away the boilerplate codes needed while using the S3Client API. For uploading smaller files and at-a-time it seems simpler to use the S3Operations. We’ll have to use the S3Client API while uploading large files and uploading in parts. Now, each file will be stored in a fixed BUCKET. However, the username is added as the prefix to store the files separately for each user within the bucket. The root BUCKET will have the following structure:
// FileController.java
/**
* Upload a file
* @param multipartFile
* @return
* @throws IOException
*/
@PreAuthorize("hasAnyAuthority('file-service:write', 'file-service:admin')")
@PostMapping("")
public ResponseEntity<?> uploadFile(@RequestParam("file") @NotNull MultipartFile multipartFile) throws IOException {
LOGGER.info(String.format("File upload request:%s, %s", multipartFile.getOriginalFilename(), multipartFile.getContentType()));
return ResponseEntity.ok().body(fileService.upload(multipartFile));
}
// FileService.java
private Supplier<String> currentUser = () -> SecurityContextHolder.getContext().getAuthentication().getName();
private Supplier<Map<String, String>> metadata = () -> {
HashMap<String, String> md = new HashMap<>();
md.put("version", "v1");
md.put("owner", currentUser.get());
md.put("timestamp", LocalDateTime.now().toString());
return md;
};
public ApiResponse upload(MultipartFile multipartFile) throws IOException {
InputStream is = new BufferedInputStream(multipartFile.getInputStream());
S3Resource upload = this.s3Operations.upload(BUCKET_NAME, buildPrefix.apply(multipartFile.getOriginalFilename()), is);
LOGGER.info("File uploaded successfully..{}", upload.getLocation());
Map<String, String> md = metadata.get();
md.putAll(Map.of("fileType", upload.contentType()));
return ApiResponse.builder().data(upload.getLocation()).status(ApiStatus.CREATED).metadata(md).build();
}- List Files: To list all the uploaded files by a user, I used the S3Client API. Unfortunately, the S3Operations API has no method for listing all the files in a given BUCKET.
// FileController.java
/**
* List all the files
* @return
*/
@PreAuthorize("hasAnyAuthority('file-service:admin','file-serive:list')")
@GetMapping("")
public ResponseEntity<?> getFileList() {
return ResponseEntity.ok().contentType(MediaType.APPLICATION_JSON).body(fileService.fileList());
}
// FileService.java
private Function<String, String> mapFileName = (key) -> key.split("/")[1];
private Function<S3Object, String> mapS3ObjectToFileName = (s3Object) -> mapFileName.apply(s3Object.key());
private Function<String, String> buildPrefix = (key) -> currentUser.get() + "/" + key;
private Predicate<String> fileExtensionFilter = (file) -> file != null;
public ApiResponse fileList() {
LOGGER.info("Fetching data from s3..");
List<String> keys = this.s3Client.listObjects(request -> request.bucket(BUCKET_NAME).prefix(currentUser.get())) // Get all the files for the user
.contents().stream().map(mapS3ObjectToFileName).filter(fileExtensionFilter) // remove the username from the file name
.collect(Collectors.toList());
return ApiResponse.builder().data(keys).status(ApiStatus.SUCCESS).metadata(Map.of("owner", currentUser.get(), "fileCount", String.valueOf(keys.size()))).build();
}- List File Version: Similar to List File API I’ll use the S3Client API for getting file versions of file. The limit is set using MAX_VERSION env. variable which is set to 20.
// FileController.java
/**
* List all the version of the file
* @param key
* @return
*/
@PreAuthorize("hasAnyAuthority('file-service:list', 'file-service:admin')")
@GetMapping("/version")
public ResponseEntity<?> getFileVersion(@RequestParam("key") @NotNull String key) {
LOGGER.info("request parma: {}", key);
return ResponseEntity.ok().contentType(MediaType.APPLICATION_JSON).body(fileService.versions(key));
}
// FileService.java
private Function<String, String> mapFileName = (key) -> key.split("/")[1];
private Function<ObjectVersion, VersionResponse> mapObjectVersionToVersionResponse = (version) -> new VersionResponse(mapFileName.apply(version.key()), version.versionId(), version.isLatest(), version.lastModified());
private Function<String, String> buildPrefix = (key) -> currentUser.get() + "/" + key;
private Supplier<String> currentUser = () -> SecurityContextHolder.getContext().getAuthentication().getName();
public ApiResponse versions(String key) {
Assert.notNull(key, "key is required");
ListObjectVersionsRequest listObjectsRequest = ListObjectVersionsRequest.builder()
.bucket(BUCKET_NAME)
.prefix(buildPrefix.apply(key))
.maxKeys(MAX_VERSION)
.build();
ListObjectVersionsResponse response = this.s3Client.listObjectVersions(listObjectsRequest);
LOGGER.info("Getting all version of {}", key);
List<VersionResponse> versionList = new LinkedList<>();
LOGGER.info("Found {} versions for {}", response.versions().size(), key);
response.versions().stream().map(mapObjectVersionToVersionResponse).forEach(vr -> versionList.add(vr));
return ApiResponse.builder()
.status(ApiStatus.SUCCESS)
.data(versionList)
.metadata(Map.of("owner", currentUser.get(), "versionCount", String.valueOf(versionList.size())))
.build();
}- Read File Content: The API will allow reading the file content optionally with the version ID. For this demo, the .txt file directly returns the content as a string, whereas any other file type will return the byte[]. However, it’s not practical to directly return the content of the file. Using a CDN like AWS CloudFront and a reference link to the file location would be a much more practical response.
// FileController.java
/**
* Read the content of the file
* @param key
* @param version
* @return
* @throws ContentTypeNotAllowedException
*/
@PreAuthorize("hasAnyAuthority('file-service:read', 'file-service:admin')")
@GetMapping("/content")
public ResponseEntity<?> getFileContent(@RequestParam("key") String key, @RequestParam("version") @Nullable String version) throws ContentTypeNotAllowedException {
LOGGER.info("Request content {}", key);
return ResponseEntity.ok().contentType(MediaType.APPLICATION_JSON).body(fileService.read(key, version));
}
// FileService.java
private Function<String, String> buildPrefix = (key) -> currentUser.get() + "/" + key;
private Supplier<String> currentUser = () -> SecurityContextHolder.getContext().getAuthentication().getName();
private Map<String, Function<ResponseBytes<GetObjectResponse>, ApiResponse.ApiResponseBuilder>> fileHandlerMap;
public FileService(S3Client s3Client, S3Operations s3Operations) {
this.s3Client = s3Client;
this.s3Operations = s3Operations;
this.fileHandlerMap = Map.of("text/plain", handleTextFile, "image/png", handlePNGFile);
}
private Function<ResponseBytes<GetObjectResponse>, ApiResponse.ApiResponseBuilder> handleTextFile = (responseBytes) -> {
ApiResponse.ApiResponseBuilder apiResponseBuilder = ApiResponse.builder();
apiResponseBuilder.data(new String(responseBytes.asByteArray(), StandardCharsets.UTF_8));
return apiResponseBuilder;
};
private Function<ResponseBytes<GetObjectResponse>, ApiResponse.ApiResponseBuilder> handlePNGFile = (responseBytes) -> {
ApiResponse.ApiResponseBuilder apiResponseBuilder = ApiResponse.builder();
apiResponseBuilder.data(responseBytes.asByteArray());
return apiResponseBuilder;
};
private ResponseBytes<GetObjectResponse> readWithVersion(String key, @Nullable String version) {
GetObjectRequest.Builder getObjectRequestBuilder = GetObjectRequest.builder();
getObjectRequestBuilder
.bucket(BUCKET_NAME)
.key(buildPrefix.apply(key));
if (version != null) getObjectRequestBuilder.versionId(version);
ResponseBytes<GetObjectResponse> response = s3Client.getObjectAsBytes(getObjectRequestBuilder.build());
LOGGER.info("Object response {}", response.toString());
return response;
}
public ApiResponse read(String key, @Nullable String version) throws ContentTypeNotAllowedException {
Assert.notNull(key, "Key is required");
ResponseBytes<GetObjectResponse> response = readWithVersion(key, version);
final String FILE_TYPE = response.response().contentType();
if (fileHandlerMap.containsKey(FILE_TYPE) == false) throw new ContentTypeNotAllowedException();
return fileHandlerMap.get(FILE_TYPE)
.apply(response)
.status(ApiStatus.SUCCESS)
.metadata(Map.of("version", "v1", "fileType", FILE_TYPE, "owner", currentUser.get()))
.build();
}- Delete File: Deleting an object from S3 is trickier. If the bucket has its versioning enabled you have manually delete every versions of the Object. Neither S3Client or S3Operations provide a deleteAllVersions or methods similar to this. Hence you have to iterate through the versions and delete each version individually. Here, the API takes an optional versionId. If provided then only that specific version is deleted.
// FileController.java
/**
* Delete file permanently
* @param key
* @param version
* @return
*/
@PreAuthorize("hasAnyAuthority('file-service:delete', 'file-service:admin')")
@DeleteMapping("")
public ResponseEntity<?> deleteFile(@RequestParam("key") String key, @RequestParam("version") @Nullable String version) {
return ResponseEntity.ok().contentType(MediaType.APPLICATION_JSON).body(fileService.delete(key, version));
}
// FileService.java
private Supplier<String> currentUser = () -> SecurityContextHolder.getContext().getAuthentication().getName();
private String deleteWithVersion(String key, @Nullable String version) {
DeleteObjectResponse response = s3Client.deleteObject((request) -> request.bucket(BUCKET_NAME).key(buildPrefix.apply(key)).versionId(version).build());
LOGGER.info("Deleting file {} {}", key, response.versionId());
return response.responseMetadata().toString();
}
public ApiResponse delete(String key, @Nullable String version) {
Assert.notNull(key, "key is required");
Map<String, String> metadata = new HashMap<>();
metadata.put("version", "v1");
metadata.put("owner", currentUser.get());
metadata.put("timestamp", LocalDateTime.now().toString());
ApiResponse.ApiResponseBuilder apiResponseBuilder = ApiResponse.builder();
apiResponseBuilder.status(ApiStatus.DELETE);
if (version != null) {
metadata.put("versionCount", String.valueOf(1));
return apiResponseBuilder.data(deleteWithVersion(key, version)).metadata(metadata).build();
}
ListObjectVersionsResponse response = null;
String nextKeyMarker = null;
String nextVersionMarker = null;
int totalVersionCount = 0;
List<String> deletedVersions = new LinkedList<>();
do {
ListObjectVersionsRequest listObjectVersionsRequest = ListObjectVersionsRequest.builder()
.bucket(BUCKET_NAME)
.prefix(buildPrefix.apply(key))
.keyMarker(nextKeyMarker)
.versionIdMarker(nextVersionMarker)
.maxKeys(MAX_VERSION).build();
response = this.s3Client.listObjectVersions(listObjectVersionsRequest);
totalVersionCount += response.versions().size();
response.versions()
.stream().map(vr -> vr.versionId())
.forEach(vid -> {
deletedVersions.add(vid);
LOGGER.info("{}", deleteWithVersion(key, vid));
});
nextKeyMarker = response.nextKeyMarker();
nextVersionMarker = response.nextVersionIdMarker();
} while (response.isTruncated());
metadata.put("versionCount", String.valueOf(totalVersionCount));
return ApiResponse.builder().data(deletedVersions).status(ApiStatus.DELETE).metadata(metadata).build();
}This concludes the Part-1. The implementation is available at my github. I hope this was helpful. I’ll we posting Part-2 as soon as I’m done implementing 4&5.
