Problem Statement

The traditional source of truth for applications is DBMS(SQL/NoSQL) and they rely on ORMs which natively support searching, sorting, filtering, pagination and more. As technologies evolved, applications moved beyond DBs and started to rely on APIs(typically REST endpoints) of other SaaS application. We all know the brighter side of SaaS offering but quite often these offering comes with numerous pitfalls, being:

• Metered/usage based pricing - Most SaaS services are priced based on usage. To cut cost, engineers typically delay usage of such offerings to the right side stages of SDLC, leading to lose of such benefits during the early stage testing.

• Vanilla implementations - As the complexity and requirements of today’s applications differ from each other, SaaS providers stick to vanilla API implementations and only based on the usage and requirements they implement features like searching, sorting, filtering, pagination that are commonly available in most ORM.

• Additional/Competitive offerings - Another common observation that we’re seeing in modern applications is that instead of relying on one datasource they need to consider multiple sources to aggregate and consolidate the data. On the other hand, when there is an alternate system offering better features, it should be easy to switch to a new source because of business demands.

To overcome these pitfalls, developers are adopting intuitive, and at certain instances, old school approaches to tackle problems of this kind. Either way, one needs to spend time building features that are readily available in ORMs. We ourselves have come across similar situations more than a couple of times and decided to devote our time in solving this problem by building a library and open source it.

TechConative built an open source library inmemory-posf and published it, inmemory-posf would save developer effort and time, allowing them to concentrate on building modern, feature-rich applications.

Features

By now you would have guessed the primary features for this library are searching, ordering, filtering, and of-course pagination, part of the obvious features that’s been highlighted in the problem statement.

Some more features of this library:

• Supports nested objects
• Pagination with limits and offset
• Case-insensitive search
• In-memory
• Light weight
• Easy to use
• Continuous Assurance Platform Integration

1. Identify SaaS datasource model class

Let’s assume you have a social media as your datasource with the following json as response.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

[
  {
    "id": 2135,
    "title": "sdf",
    "description": "sdf",
    "multiMedia": [
      {
        "id": 3235,
        "name": "",
        "url": "https://www.youtube.com/watch?v=4uY4Pz0SuaM",
        "likeCount": 0,
        "createAt": "0001-01-01T00:00:00"
      }
    ],
    "likeDislike": {
      "likes": 0,
      "dislikes": 0
    },
    "createdAt": "2020-01-02T10:35:11.7588502",
    "msg": null
  }
]

The following POJO could be it’s corresponding model class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

public class Feed {

    public int id;
    public String title;
    public String description;
    public List<MultiMedium> multiMedia;    //nested object
    public LikeDislike likeDislike;         // one more nested object
    public Date createdAt;
    public String msg;
}

2. Import the library into your project

maven

1
2
3
4
5
6

<!-- pom.xml -->
<dependency>
  <groupId>com.techconative</groupId>
  <artifactId>inmemory-posf</artifactId>
  <version>1.0.0</version>
</dependency>

gradle

1
2
3
4

// build.gradle
dependencies {
  implementation 'com.techconative.inmemory-posf:1.0.0'
}

3. Define the data source - Extend POSFService class from the library and override getRawData() method

getRawData() supplies the input data to our library

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

public class FeedService extends POSFService<Feed> {

    private static ObjectMapper mapper = new ObjectMapper();

    @Override
    protected List<Feed> getRawData() {
        try {
              //for simplicity assume feed data is stored in a file.
              return mapper.readValue(
                      this.getClass()
                              .getClassLoader()
                              .getResourceAsStream("feed-data.json"),
                      new TypeReference<List<Feed>>() {});
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }
}

4. Define your criteria

Now we’re all set to use the library, refer our GitHub repository for the criteria syntax.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

    POSFCriteria criteria = new POSFCriteria();

    // 1.  Filtering and search
    criteria.setFilter("multiMedia.[].name=CCCC&*=Vega|vegas&userId=4051");

    /* 2. Sort filtered data
     * Column name to sort by
    */ 
    criteria.setColumn("id");
    // Sorting criteria. Accepts ASC / DESC.
    criteria.setSort(OrderingCriteria.ASC);

    /* 3. Paginate sorted data
     * Number of items that need to to displayed in a page. 
     * If it is 0 then it will return all records.
    */  
    criteria.setLimit(10);

    // 4. Offset page to be returned after paginating the data.
    criteria.setPageNumber(1);

5. Apply criteria

1
2

IPOSFService service = new FeedService();
PageResult pageResult = service.getPageResult(criteria);

6. Access your processed data

1

List<Feed> resultData = pageResult.getData();

pageResult.getData() returns the processed dataset as per the criteria defined.

In case there are limitations in your design and not able to extend POSFService don't worry we have do have alternate approaches for you. Tweak Step #3 & Step #5 as the following

3. Define the data source (alternate) - Fetch your data and store as a List.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

List rawData = null;

try {
      rawData = mapper.readValue(
              this.getClass()
                      .getClassLoader()
                      .getResourceAsStream("feed-data.json"),
              new TypeReference<List<Feed>>() {});
} catch (IOException e) {
    throw new RuntimeException(e);
}

5. Apply criteria (alternate) - Invoking library as a utility

1

PageResult pageResult = POSFUtil.processData(rawData, criteria);

Thoughts for future features

We have identified few features that could augment the current feature set:

• ORM for SaaS service
• Regex support

Feel free to let us know your thoughts and feature suggestions that you would like built. We would be elated if you join us in building this library further. Meanwhile, You can also look through and contribute to the source in our git repository.