Searching, Sorting and Pagination in a Common Lisp web application

As usual we are going to use Caveman for scaffolding our web application. If you want to know more about Caveman or why Caveman, you might want to take a look at my previous post about Common Lisp and web development. Caveman is available in quicklisp, so you can install it with:

(ql:quickload :caveman2)

And you can start a create a new project with Caveman like this:

(caveman2:make-project #P"~/quicklisp/local-projects/cl-tabular" :author "Rajasegar")

Now take a look at our index route. Our index route is going to use some query params for sorting and pagination. With Caveman you can parse the query params using the _parsed key and we are using custom defined function to take the query parameter values from it like below:

(defun query-param (name parsed)
  (cdr (assoc name parsed :test #'string=)))

So for our route logic, we need the values of the following query parameters, start which is the starting offset of the records for the page, direction which is either ascending or descending and the sort-by key based on which column we are currently sorting the list.

(defroute "/" (&key _parsed)
  (format t "_parsed = ~a~%" _parsed)
  (let ((start (parse-integer (or (query-param "start" _parsed) "0")))
        (direction (or (query-param "direction" _parsed) "asc"))
        (sort-by (or (query-param "sort-by" _parsed) "name")))
    (render #P"index.html"
            (list
             :foods (slice-list start (sort-list direction sort-by))
             :total (length *foods*)
	     :pages (generate-pages)
	     :start start
	     :direction direction
	     :sort-by sort-by
	     :opposite-direction (get-opposite-direction direction)))))

One thing important to note in the above code snippet is how we are sending data to the templates via the render function. We can construct our data using a list and the templates can easily access them via the keyword mapped to the data.

Say, for example, in the template you can loop through the list of foods like

{% for food in foods}
<p>{{food.name}}</p>
{% endfor %}

So how are we sending the paginated list of foods to our home page. If you take a closer look, we are first sorting the entire list of foods using the sort-by parameter and then we are slicing the list based on the start offset and returning them to the template.

:foods (slice-list start (sort-list direction sort-by))

Let's take a look at our slice-list function on how we are slicing our list.

(defun slice-list (start)
  (let ((new-list nil))
    (dotimes (i 10)
      (push (elt *foods* (+ i start)) new-list))
    new-list))

We construct a new temporary list by pushing the 10 items from the original list, starting from the start offset and then returning the new list to the page. The sort-list function is discussed in the later part of the post under Sorting.

Our index template is very big and has got three sections, the search form, the table and the pagination.

  <form>
    <div class="mb-4">
      <div class="col-6">
        <input 
            class="form-control form-control-lg" 
            type="text" 
            placeholder="Search dish name..."  
            name="query" 
            hx-post="/search?start=0&direction=asc&sort-by=name" 
            hx-trigger="keyup changed delay:500ms" 
            hx-target="#results">
      </div>
    </div>
  </form>

  <div id="results">
    <p>{{total}} results found</p>
    <table class="table table-striped">
      <thead>
        <tr class="table-dark">
          <th><a href="/?start=0&sort-by=name&amp;direction={{opposite-direction}}">Name
	      {% if sort-by == "name" and  direction == "asc" %} &uarr; {% endif %}
	      {% if sort-by == "name" and  direction == "desc" %} &darr; {% endif %}
	  </a></th>
          <th><a href="/?start=0&sort-by=rating&amp;direction={{opposite-direction}}">Rating
	      {% if sort-by == "rating" and  direction == "asc" %} &uarr; {% endif %}
	      {% if sort-by == "rating" and  direction == "desc" %} &darr; {% endif %}
	  </a></th>
          <th><a href="/?start=0&sort-by=price&amp;direction={{opposite-direction}}"> Price
	      {% if sort-by == "price" and  direction == "asc" %} &uarr; {% endif %}
	      {% if sort-by == "price" and  direction == "desc" %} &darr; {% endif %}
	  </a></th>
          <th><a href="/?start=0&sort-by=cuisine&amp;direction={{opposite-direction}}">Cuisine
	      {% if sort-by == "cuisine" and  direction == "asc" %} &uarr; {% endif %}
	      {% if sort-by == "cuisine" and  direction == "desc" %} &darr; {% endif %}
	  </a></th>
        </tr>
      </thead>
      <tbody>
        {% for food in foods %}
        <tr>
          <td>{{food.name}}</td>
          <td>
	    {% ifequal food.rating 1 %}&starf;{% endifequal %}
	    {% ifequal food.rating 2 %}&starf;&starf;{% endifequal %}
	    {% ifequal food.rating 3 %}&starf;&starf;&starf;{% endifequal %}
	    {% ifequal food.rating 4 %}&starf;&starf;&starf;&starf;{% endifequal %}
	    {% ifequal food.rating 5 %}&starf;&starf;&starf;&starf;&starf;{% endifequal %}
	  </td>
          <td>
	    ${{food.price}}
	  </td>
          <td>{{food.cuisine}}</td>
        </tr>
        {% endfor %}
      </tbody>
    </table>
  </div>

    <nav aria-label="Page navigation example">
      <ul class="pagination">
	{% for page in pages %}
        <li class="page-item {% ifequal start page.start %} active {% endifequal %}" >
	  <a class="page-link" href="/?start={{page.start}}&amp;direction={{direction}}&amp;sort-by={{sort-by}}">{{page.id}}</a>
	</li>
	{% endfor %}
      </ul>
    </nav>

(defun generate-pages ()
  "Generate pagination"
  (let ((pages nil))
    (dotimes (i 10)
      (push (list :id (+ 1 i) :start (* 10 i)) pages))
    (reverse pages)))

The data for our table is just a random list of dishes, ratings, price and the cuisine. First we declare a global variable called *foods* and initialize the value to nil.

(defvar *foods* nil)

(defvar *dishes* '("Pizza"
		   "Noodles"
		   "Fried Rice"
		   "Roti"
		   "Lasagna"
		   "Churros"
		   "Tea"
		   "Soup"
		   "Egg roll"
		   "Salad"
		   "Burger"
		   "Rice"
		   "Curry"
		   "Bread"))

(defvar *cuisines* '("Indian"
		     "Chinese"
		     "Thai"
		     "Continental"
		     "Mexican"
		     "Indonesian"
		     "Japanese"
		     "Spanish"
		     "Italian"
		     "Greek"))

Now it's time to combine all our dish names and cuisines to generate a list of dishes with random rating values and prices. So before pushing the generated values into our global foods variable, let's be sure to reset the variable to nil.

Then using a dotimes loop for 100 iterations we are going to generate a random record for dish. We are getting a random dish and cuisine form the previously created lists called dishes and cuisines respectively.

;; Clear the list
(setf *foods* nil)

;; Push 100 items into foods with random values
(dotimes (i 100)
  (push (list :name (random-elt *dishes*)
	      :cuisine (random-elt *cuisines*)
	      :rating (+ 1 (random 5))
	      :price (+ 1 (random 100))) *foods*))

For that we are using a custom defined function called random-elt which will pick a random element from a list.

(defun random-elt (mylist)
  (elt mylist (random (length mylist))))

And then for the rating and price, we are using the standard library function called random to generate random numbers within a specified range. For example, (random 5) will generate random numbers between 0 and 4 and we are adding 1 to ensure we are getting a non-zero value.

Sorting data in Common Lisp is pretty easy and straight-forward when it comes to lists. We are using an higher-order function called sort-list which will take two parameters, the sort direction either "asc" or "desc" and the sort-by which is the key based on which we sort the list. And based on the sort-by key we will delegate the sorting to the respective sort functions with the direction as an argument.

(defun sort-list (direction sort-by)
  "Sort a list based on the direction and key"
  (cond ((string= sort-by "name") (sort-list-by-name direction))
	((string= sort-by "rating") (sort-list-by-rating direction))
	((string= sort-by "price") (sort-list-by-price direction))
	((string= sort-by "cuisine") (sort-list-by-cuisine direction))))

Based on the direction, we will figure out the sort function to use, #'string> or #'string< for name and cuisine, and #'> or #'< for rating and price. We can still have one function for sorting all the columns if we can refactor, becuase this approach will not scale for large number of columns in the table.

(defun sort-list-by-name (direction)
  "Sort a list by name"
  (let ((sort-fn (if (string= direction "asc") #'string< #'string>)))
    (sort (copy-list *foods*) sort-fn :key (lambda (plist) (getf plist :name)))))

(defun sort-list-by-rating (direction)
  "Sort a list by rating"
  (let ((sort-fn (if (string= direction "asc") #'< #'>)))
    (sort (copy-list *foods*) sort-fn :key (lambda (plist) (getf plist :rating)))))

(defun sort-list-by-price (direction)
  "Sort a list by price"
  (let ((sort-fn (if (string= direction "asc") #'< #'>)))
    (sort (copy-list *foods*) sort-fn :key (lambda (plist) (getf plist :price)))))

(defun sort-list-by-cuisine (direction)
  "Sort a list by price"
  (let ((sort-fn (if (string= direction "asc") #'string< #'string>)))
    (sort (copy-list *foods*) sort-fn :key (lambda (plist) (getf plist :cuisine)))))

Next we focus on the search route for our application. The search route will take a query parameter called query itself, through which we will get the search keywords for the route. We will perform the search only based on the names of the dishes. We will use a utility function called filter-foods for this purpose.

(defroute ("/search" :method :POST) (&key _parsed)
  (format t "_parsed = ~a~%" _parsed)
  (let* ((query (cdr (assoc "query" _parsed :test #'string=)))
        (filtered-foods (filter-foods query)))
    (render #P"_search.html"
            (list
             :foods filtered-foods
             :total (length filtered-foods)))))

(defun filter-foods (query)
  "Filter foods based on the query with name"
  (remove-if #'(lambda (food)
                 (let ((name (getf food :name)))
                   (if (search query name :test #'char-equal)
                       nil
                       t))) *foods*))

<div id="results" >
  <p><a href="/">Clear Search</a></p>
  <p>{{total}} results found</p>
  <table class="table table-striped">
    <thead>
      <tr class="table-dark">
        <th> <a href="/?sort-by=name&amp;direction=desc"> Name ↓</a></th>
        <th> <a href="/?sort-by=stars&amp;direction=desc"> Stars</a></th>
        <th> <a href="/?sort-by=price&amp;direction=desc"> Price</a></th>
        <th> <a href="/?sort-by=category&amp;direction=desc"> Category</a></th>
      </tr>
    </thead>
    <tbody>
      {% for food in foods %}
      <tr>
        <td>{{food.name}}</td>
        <td>{{food.rating}}</td>
        <td>{{food.price}}</td>
        <td>{{food.cuisine}}</td>
      </tr>
      {% endfor %}
  </tbody></table>
</div>

And this is how the final app look like in action.

The source code for this application is hosted in Github. If you are stuck with any step or anything is missing in this post, you can always refer to the updated source code in Github.