Whilst building my tutorial in Cross-platform mobile app prototyping with Flutter and AWS Amplify, I encountered a few issues whilst deploying an API capable of storing user details. My requirements were fairly basic – store a few simple details about a user (name, location and favourite programming language), and query this information based on the logged in user of my app.
In Amplify, you can very quickly build and deploy and API with a single command:
amplify add apiThe following may feel like I’m taking you on a wild goose chase, but I figure if I shared the journey and pain encountered along the way you’ll understand and avoid the limitations without spending the time I did figuring it all out!
Anyway, back to our original programming…. Amplify attempts to default the questions asked when creating an API, but you can override any of these by scrolling up to the option you want to change. In this scenario we want a GraphQL endpoint (to leverage DataStore), and the API authorisation to be handled by the Cognito User Pool, as we want to apply fine-grained access control so only the user can update or delete their own details (but other users can view them).
When we create the API, Amplify creates a basic ToDo GraphQL schema type. We’ll update this and add some authorisation rules before pushing the api changes.
Modify the “ToDo” template GraphQL schema to cater for our User Profile information:
https://docs.amplify.aws/cli/graphql/authorization-rules/#authorization-strategies
type UserProfile @model @auth(rules: [
{ allow: private, operations: [read], provider: iam },
{ allow: owner, operations: [create, read, update] }
]) {
name: String!
location: String
language: String
}At this point we can provision our cloud infrastructure supporting the API feature:
amplify pushSo once we’ve defined a GraphQL schema, we can use Amplify to generate the Dart code that represents the model and repository layer that can work with the API:
amplify codegen modelsWhen you create objects and save them via AppSync, additional metadata fields get attached to them when persisted in DynamoDB:
{
"id": "5e62567e-52fd-4793-b9cb-96dc36baffef",
"createdAt": "2023-01-18T11:17:54.595Z",
"language": "Java, Dart & Flutter",
"location": "Leeds",
"name": "Ben Foster",
"owner": "80842775-2bad-4647-a1e5-1b107a4e8bdf::80842775-2bad-4647-a1e5-1b107a4e8bdf",
"updatedAt": "2023-01-18T11:17:54.595Z",
"userId": "80842775-2bad-4647-a1e5-1b107a4e8bdf",
"__typename": "UserProfile"
}The two fields of interest here are the id and owner fields. These are autogenerated by AppSync in its resolvers and persisted along with your object in DynamoDB. The owner field stores the sub of the Cognito User, and is used by the authentication directives (the @auth directive in the GraphQL schema) to filter out objects that users should or shouldn’t see.
Rabbit Hole #1 – When you persist objects in AppSync, it assigns an autogenerated UUID as its id. The way you’re supposed to find records is by using either ModelQueries.list or ModelQueries.get and then passing the id of the record. But we actually want to look up by the user who created it. When creating the above schema, Amplify uses the id field as the partition key in the DynamoDB table. In my particular data model I’d rather have the owner be the partition key, as that is what people will query by.
TL;DR: Read on to #Rabbit Hole 3 ahead and use a combination of creating a separate userId property to track the owner (with an @index directive applied), and query directly using a GraphQL query as opposed to the high-level AppSync API.
Trying to tidy up the DynamoDB table so there aren’t separate id and owner fields, but make the owner field the partition key (which also enforces a single entry per user)
Apparently this isn’t advised:
https://github.com/aws-amplify/amplify-category-api/issues/809
https://github.com/aws-amplify/amplify-category-api/issues/229
Trying this as a workaround, setting ownerField to be “id” then applying field level permissions to restrict modification
Trouble with this is you encounter code generation issues if you have other required fields on the model, you have to explicitly give them at least read permission (which I didn’t want to do)
I suspect this isn’t a supported use case, and something in the generated createUserProfile mutation resolver is causing this to happen
You’d need to create a custom resolver to inject the users identifier at API level into the partition key, but that takes away from Amplify’s simplicity, and out of scope for this tutorial. For now we’ll just query by owner and assume there’s only one UserProfile record per user.
Rabbit Hole #2 – It appears that you aren’t able to query by the owner field – it’s an internal field that shouldn’t be used directly, and only by the framework to enforce row-level security.
There’s been a recent announcement showcasing the ability to specify a custom primary key, which we could use to create a separate userId field used by us that stores the Cognito identity sub and set that as the partition key.
https://aws.amazon.com/blogs/mobile/new-announcing-custom-primary-key-support-for-amplify-datastore/
I tried using @primaryKey directive as below, unfortunately this doesn’t work because support for the directive hasn’t reached the amplify_api dart library yet. It assumes that the id field is always called “id” (see model_queries_factory.dart:32, drill into idFieldName and you’ll see the comment saying this should eventually be dynamic)
type UserProfile @model
@auth(rules: [
{ allow: private, operations: [read], provider: iam },
{ allow: owner, operations: [create, read, update, delete] }
])
{
userId: String! @primaryKey
name: String!
location: String
language: String
}If you try and remove the default id field, and set owner as the @primarykey then the table is updated to reflect but when you try to save you get this error:
flutter: errors: [GraphQLResponseError{
"message": "Not Authorized to access createUserProfile on type UserProfile",
"locations": [
{
"line": 1,
"column": 108
}
],
"path": [
"createUserProfile"
],
"extensions": {
"errorInfo": null,
"data": null,
"errorType": "Unauthorized"
}
}]A nicer solution would be to manually override the AppSync resolvers to set the id field to that of the user’s Cognito Id when saving. Delving into AppSync, GraphQL, resolvers and VTL (Apache Velocity Template Language) is out of the intended scope of this blog, but maybe one for another day as it’s something I’d like to see possible with Amplify.
For now, what we’ll do is create a separate userId field with an @index directive, which creates a global secondary index (GSI) in DynamoDB to move us from our inefficient table scans to more efficient index lookups. When querying in code we can also add a predicate for userId when loading details from AppSync. This way we’re also ensuring there’s only one per user (as with an autogenerated id we could potentially have duplicates if there’s a bug somehwere) and not relying solely on the row-level security to return the right record:
type UserProfile @model
@auth(rules: [
{ allow: private, operations: [read], provider: iam },
{ allow: owner, operations: [create, read, update, delete] }
])
{
userId: String! @index
name: String!
location: String
language: String
} void _getUserProfile() async {
final currentUser = await Amplify.Auth.getCurrentUser();
final queryPredicate = UserProfile.USERID.eq(currentUser.userId);
final request = ModelQueries.list<UserProfile>(UserProfile.classType,
where: queryPredicate);
final response = await Amplify.API.query(request: request).response;So at this point, we have a means of querying, displaying, and updating a user’s profile information. Feels like another save point to me.
Rabbit Hole #3 – Always wanting to understand how “magic” things work, I peeked behind the covers and analysed the types of requests being thrown at DynamoDB when looking up user profiles. Turns out that even though we’d applied the @index directive to the userId field and included it as a predicate in our request, AppSync was still calling our table with a scan and not a query, using this filter expression:
"filterExpression": "(((#owner = :and_0_or_0_owner_eq) OR (#owner = :and_0_or_1_owner_eq) OR (#owner = :and_0_or_2_owner_eq)) AND (#userId = :and_1_userId_eq))"This makes me sad.
I had hoped similar to using the above combination with DataStore and Sync Expressions, this would coax Amplify to query by the userId GSI, but alas this didn’t happen. I’m unsure if this is a limitation of the Flutter library or Amplify overall.
The way in which AppSync interacts with our DynamoDB table is governed by the GraphQL queries, and the AppSync resolvers that sit inbetween the incoming GraphQL queries and the data source. Our code behind the scenes is calling the listUserProfiles or getUserProfiles queries (depending on if you ask for a list or query by internal id). When you add the @index directive to the userId field, an additional resolver is created called userProfilesByUserId, which is specifically for using the GSI.
Knowing this, we can use it in our app by referring to the low-level GraphQL query in a hand-cranked GraphQL document:
void _getUserProfile() async {
final currentUser = await Amplify.Auth.getCurrentUser();
GraphQLRequest<PaginatedResult<UserProfile>> request = GraphQLRequest(
document:
'''query MyQuery { userProfilesByUserId(userId: "${currentUser.userId}") {
items {
name
location
language
id
owner
createdAt
updatedAt
userId
}
}}''',
modelType: const PaginatedModelType(UserProfile.classType),
decodePath: "userProfilesByUserId");
final response = await Amplify.API.query(request: request).response;So the takeaway from all of this is whenever you’re building a prototype, always assess from an NFR perspective whether the solution will scale up to production level
I’ve tried to summarise the various options tried above, with pros and cons:
| Option | Pros | Cons |
|---|---|---|
| Option 1 – Create separate “userId” field and use @index directive to create a GSI | Efficient querying with DataStore and Sync Expressions – avoids full table scans Can use high-level Amplify.API method to save Authorisation and query attributes are decoupled (although could argue that’s unnecessary duplication) | App is responsible for populating userId, can be exploited by malicious app code or MITM Redundant id field as partition key, as we’ll always query by userId Need to use lower-level GraphQL API to query by resolver for userId GSI |
| Option 2 – Create custom GraphQL resolvers that inject the users id from credentials in the request into the “id” field | Can use high-level Amplify.API method to save Authorization directives are unaffected | Retrieval by id still requires the mobile app passing the users id (but as a query that’s maybe not necessarily a bad thing) Requires knowledge of resolvers and either VTL or hooking up a lambda resolver to implement a custom resolver |
| Option 3 – Rely on authorisation mechanism to only return user profiles created by this user (should always be 1) | Can use high-level Amplify.API method to save No need for any custom resolver config or leaking abstractions from the GraphQL API | Inefficient table scan of all UserProfiles, to then query by ones where the owner matches Won’t allow you to query other user profiles |
| Option 3.1 – Separate resolver to work with user’s specific profile, and another to query other peoples profiles | Can use high-level Amplify.API method to save Authorization directives are unaffected | Requires a separate custom resolver for querying other people’s profiles. Logic would have to replicate the “private” authorisation rule |
Feels like best option long term is to wait until the @primaryKey directive is fully implemented, but unsure of an ETA or what the migration path to using that would be.
Rabbit Hole #4 – Offline and Realtime updates with Amplify DataStore
For the icing on the cake I wanted to demo Amplify DataStore, which allows you to cache data offline on your device, make changes and then automatically sync once reconnected to the internet. This is great for scenarios where internet connectivity isn’t guaranteed (e.g. remote site workers).
However it wasn’t as straightforward as I hoped, and there appears to be a bug preventing my particular model from correctly persisting in the app’s SQLite database.
Here’s the AWS Amplify docs on how to integrate Amplify DataStore with Flutter: https://docs.amplify.aws/lib/datastore/real-time/q/platform/flutter/
The first issue I encountered was needing to change the minimum target API version for iOS, because DataStore only support iOS 13 and higher (Flutter by default supports 11+)
Launching lib/main.dart on iPhone 14 Pro Max in debug mode...
CocoaPods' output:
...
Error output from CocoaPods:
↳
[!] Automatically assigning platform `iOS` with version `11.0` on target `Runner` because no platform was specified. Please specify a platform for this target in your Podfile. See `https://guides.cocoapods.org/syntax/podfile.html#platform`.
Error running pod install
Error launching application on iPhone 14 Pro Max.
ExitedNeed to specify minimum target API for iOS (similar to what we did with Android earlier on)
Uncomment the second line in ios/Podfile and change platform :ios, '11.0' to platform :ios, '13.0'
Needed to run an amplify update api to enable conflict resolution, followed by an amplify push.
But alas this also wasn’t enough. Logs in the `Debug Console` in VSCode coming from XCode build weren’t overly helpful, so I switched over to my Android emulator. The logs there were much more useful (I’m guessing there’s some config I need to increase verbosity with XCode logs).
Few exceptions thrown on startup relating to the DataStore, this log message in particular I think is an indicator
Orchestrator transitioning from SYNC_VIA_API to LOCAL_ONLY
Definitely don’t want that, so what’s causing it? There’s a few exceptions higher up the logs:
Caused by: DataStoreException{message=Failure performing sync query to AppSync: [GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'AWSTimestamp' within parent 'UserProfile' (/syncUserProfiles/items[0]/_lastChangedAt)'
This exception seems to be thrown a few times, but what I think this is saying is that we can’t just configure a DataStore when data already exists in the backing store (at least not in the OOTB configuration of DataStore). Because our UserProfile hasn’t been connected to a DataStore before, there’s some mandatory properties (i.e. _lastChangedAt) that don’t exist in DynamoDB. Let’s clear out the UserProfile record for my account and see if that fixes things…
Actually what happens if we just add the expected `_lastChangedAt` value to our existing UserProfile record? Well that field is of type AWSTimestamp, which in AppSync translates to an integer field with the number of milliseconds since epoch.
Well, the above got us a little further, on to the next exception that is:
Caused by: DataStoreException{message=Failure performing sync query to AppSync: [GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'Int' within parent 'UserProfile' (/syncUserProfiles/items[0]/_version)'
Following the same approach and adding a _version attribute to our profile record did the trick. So to convert an existing AppSync DynamoDB backed datastore to support DataStore, you need to add a `_lastChangedAt` and `_version` attribute and default them to some sensible values that work for you (for this demo I just set it to the current date time and 1 respectively)
I would’ve hoped that Amplify would handle this migration path more gracefully… let’s see if there’s an issue raised around this already and report it if not…
Now let’s see if I can change the values on one device and have them magically update in another…
Next problem is I can’t actually update the UserProfile, because SQLite for some reason is trying to do an INSERT into the local device database, and failing on the userId unique constraint (created by the @index directive that triggers the GSI being created). It seems to be trying to create a new UserProfile entry, despite me using the same internal id to save with.
Just trying to query and then immediately save the object back into the DataStore as demonstrated in the AWS docs produces the same issue.
Going to delete the record from DynamoDB and start from scratch, see what happens… still same issue. When saving profile for the first time it syncs the new record to DynamoDB but still hits this SQLite issue, subsequent updates aren’t pushed to DynamoDB (presumably because of the SQLite issue).
So looks like we can’t save UserProfiles when using the DataStore, will raise an issue for that.
I wonder if we can at least demo the real-time update of data in the app if I directly update the record via AppSync (not DynamoDB as there’s various bits of metadata that has to be updated too, hopefully the GraphQL resolvers handle this…)
Wrapping up
I hope this deep-dive into the challenges I faced when working with Amplify and AppSync are useful for anyone else embarking on a similar journey. AppSync, GraphQL and DataStore is a very powerful tech stack, but when used in an OOTB “low code” solution like Amplify can result in sub-optimal choices being made, which can cause headaches later on (especially when they’re not utilised to their fullest potential). Always research tooling choices, and never be afraid to migrate away when they become more of a hinderance than a help.
