What's new in FastStats?

Added in Q4 23 - New penetration mapping capability

  • Penetration mapping compares particular groups of customers by location. This new functionality builds upon the capabilities of the modelling Profile tool, which allows you to find categories and characteristics of interest or over/under-representation within an analysis group and compare them to a wider base group. Now you can work in a similar way to to search for and visualise geographical insight in your data using the Map tool.

    See Bing Maps: Penetration Mapping

 

Added in Q4 23 - N% per variable sampling in selections

  • N per table (such as 1 per household) and N per variable (such as 1 per email) functionality has been available in FastStats for many years, but is now extended to provide a much simpler solution to a common sampling requirement - that is, to sample N% of records whilst maintaining the distribution across an underlying variable.

    See Selection: How do I make a N% per variable selection?

 

Added in Q4 23 - Cube, tree and word cloud conversions to map

  • For a cube, tree or word cloud which has a single geographic variable, such as Postal Area, you can right drag and convert into a map. The variable must be a selector and have the GeoFormat property set for this to be possible.

    See Cube: How do I switch between cube-based visualisations?

 

Added in Q4 23 - Change to map drivetime dialogue

  • When creating drivetime analysis, you now have two choices for selecting the origin of the drivetime - either the point you clicked, or the point you searched for. This can help to produce more accurate drivetimes, particularly when supplying the Bing Isochrone API with a centrepoint that is not near any roads.

    See Bing Maps: DriveTime Calculations

 

Added in Q4 23 - Extension to Relative Transaction on the fly aggregation

  • The ability to include a third piece of transactional information was introduced for Rank, Recency and Pattern Match aggregations in the Q3 release. This is now extended to include Relative aggregations and, like Recency, allows you to specify two ordering variables and use the second one to break ties when the first one is equal.

    See Expressions: Aggregations on the fly - Relative Transaction

 

Added in Q4 23 - New and updated expression functions

A number of new, as well as updates to existing, expression functions have been implemented.

New functions include:

 

Added in Q3 23 - Extension to on the fly aggregation capabilities

For the ranking, recency and pattern match techniques, the on the fly aggregation processing mechanism now provides the option to pass through three pieces of transactional information instead of two.

  • Pattern match - allows you to specify a grouping variable, with transactions split into these groups before the pattern match algorithm is applied to each group individually.

    See Pattern Match with Grouping

 

Added in Q3 23 - Custom colours in cube, tree and map thematics

  • In addition to all existing thematic options, you can now user define and set a custom sequence of colours for the bands displayed in a cube, tree or map. This is useful for presenting insight when the graduation of colour is not significant, but differentiating information in the display is beneficial.

    See Custom Colours in Cube, Tree and Map Thematics

 

Added in Q3 23 - Custom labels in map thematics

  • The introduction of custom colours on a map also brings the ability to set up corresponding custom labels for each of the bands in your thematic shading. These labels are visible in the map legend and help to provide further meaning to user defined colours

    See Bing Maps: Shaded Map - Custom Thematic Labels

 

Added in Q3 23 - New context sensitive help in expressions

  • New context sensitive help in the expression window provides an explanation and worked example for each function.

    See Expressions: Overview

 

Added in Q3 23 - Enhanced scheduled task error notification

  • In addition to the task owner, you can now specify a list of additional email addresses to which a notification should be sent in the event that a scheduled task errors.

    For full details of the options see Scheduled Tasks

 

Added in Q3 23 - Bing transaction logging

  • Bing transactions are now recorded and stored in CSV files, allowing system administrators to easily track, verify and manage usage for those creating map analyses and visualisations.

    See Bing API - Bing transaction logging

 

Added in Q2 23 - Modelling developments

  • Incremental Insight

    A new Incremental Insight metric provides a single measure that allows you to select dimensions which are both predictive and diverse. It is based on the Insight PWE metric, but accounts for the correlation between dimensions.

  • Chart Options

    Scatter charts can now display bubbles to show both Insight PWE (using colour) and Insight Coverage (using size).

 

Added in Q2 23 - Map improvements

  • Bing API Isochrone caching

    • Drive zones created using the mapping wizards are now stored. This means that, when editing the variable, you do not incur new Bing API transactions for zones that have previously been calculated.

    • You can configure zones to have an expiry period after which they are then recalculated.
  • BIng API geocoding caching
    • Searching for locations on an interactive map, or through specifying centrepoints in the Map wizards, incurs Bing API geocoding transactions. From Q2 2023, specifying a centrepoint in 'Latitude.Longitude' format (for example, 52.281806,-1.588657) is treated as valid location information. This is a user-configurable property.
    • If you have to request a geocoded location from the Bing API, the resulting latitude and longitude is stored and can be reused if you request it again. This is supported in the interactive map search and the map wizards Centrepoint step.

    For more information on the above, see Bing API.

  • Territories wizard

    • For the option to create and link a new shapefile to an existing selector variable, FastStats now automatically checks for categories in the variable which do not match those in the shapefile. You then have the opportunity to update them in the Unmatched Categories step.

    • Clicking on the header row in the list of shapefile features also selects that option in the drop-down list.

      See Territories Wizard: Bing Map Territories

  • The following usability improvements allow you to create maps more quickly:

    • You can now drag variables directly on to the Statistics panel on a map layer, rather than having to use the Add Statistics button.

    • Changing the primary statistic on a thematic map layer automatically updates the thematic shading without the need to rebuild the map.

      See Bing Maps: Shaded Maps - Adding Statistics

    • On a thematic map layer, you can now right click on the Geographic Variable drop-box to easily access, select and add any available selector variables with a 'GeoFormat' property set.

      See Bing Maps: Shaded Map

    • On a plot map layer, you can now right click on the Geographic Variable drop-box to easily access, select and add any selector or text variable with a 'GeoFormat' property set.

      See Bing Maps: Plot Map - Using Longitude and Latitude Co-ordinates

    • In Map wizards, where you need to specify latitude and longitude variables, you can now right click and access a menu option that allows you to select and add variables which are marked with the relevant 'GeoFormat' property.

    • When using the Location Geocoder wizard, you can now select a check-box to confirm that your geographic variable is a UK postocode. This allows the virtual variable to be generated using the UKPostcode expression rather than using Bing Geocode. The results are returned significantly faster and without incurring Bing transaction costs.

      See Location Geocoder: Overview

 

Added in Q2 23 - Import Data wizard descriptions from column in file

In the Import Data wizard you can choose to create a selector type variable from the data file. Previously, both the codes and descriptions for the resulting variable were taken from the data column and had to be the same. From Q2 2023, you can specify a second column in the data file and use this to generate the category descriptions for the resulting variable.

See Import Data Wizard

 

Added in Q2 23 - New and updated expression functions

A number of new, as well as updates to existing, expression functions have been implemented.

New functions include:

  • Selector Functions

    • VarSelectNth - used to return an integer of the Nth code from the list that matches the code of the selector variable category

    • VarSelectCodeNth - used to return the Nth matched code based on each selector variable being checked against its own separate list of match codes

  • DateTime List Functions

    • FirstDateTime/LastDateTime - used to return the earliest or latest datetime from a list of datetimes

    • NthDateTime - returns the Nth datetime specified from a list of datetimes ordered from earliest to latest

 

Added in Q1 23 - Modelling developments

The selection of behavioural features has the greatest impact upon the final quality of a data model. Whilst recent development has focussed on charts and metrics which help you to assess existing features in order to identify subsets which might be predictive or diverse, this release makes it easier to generate alternative behavioural features. This includes:

A further significant development is:

 

Added in Q1 23 - Ability to reference selector and numeric values to use in a selection

Whilst text variables have always supported an option to reference a file of text values to use in a selection, this is now also supported for selector and numeric variables.

  • Numeric variables are supported as a set of ranges.

  • For selector variables there is a new option to reference the codes in the file at the time a selection is actually run. This is in addition to the pre-existing import option which takes the current values in a reference file and matches those current values to the relevant codes in order to select them.

    See Selection: How do I import values by reference?

 

Added in Q1 23 - Map wizard improvements

  • Drive Zone wizard

    When creating drive zones, in addition to the existing (now default) selector type virtual variable, you can also create a single multi-response (flag array) virtual variable from all of the centrepoints.

    See Drive Zone Wizard: Overview

  • Territories wizard

    It is now possible to see a sample of the selected shapefile's features and values, allowing you to more easily select a sensible field for the resulting variable's categories.

    See Territories Wizard: Json File Details

  • Removal of Microsoft MapPoint

    In the Q4 22 release notes we announced that we would cease support of MapPoint from Q1 23. As a result:

    • The Settings page in the Drive Zone and Point to Point wizards has been removed

    • You no longer need to specify World region and Preferred route in the FastStats Tools-Options

    • You can no longer select to use Routes API as a drivetime option in the Map tool

 

Added in Q1 23 - Ability to set GeoFormat via Expression wizard

Recent developments in the mapping wizards often require the use of latitude and longitude variables . Where such variables are not already present in your system data, you can use the UKPostcode expression function to turn a postcode into a latitude or longitude reference. In such cases, the Choose Data Type step of the Calculate Expression wizard now also allows you to set the GeoFormat on these virtual variables so that you can then use them in the mapping wizards.

 

Added in Q1 23 - New and updated expression functions

A number of new, as well as updates to existing, expression functions have been implemented.

New functions include:

  • NthDate - this Date List Function complements the existing FirstDate and LastDate functions and returns the Nth date in a list of dates

  • NthDistinct and NthDistinctIndex - two new Sorting Functions which sort numbers, dates or datetimes into ascending sequence and then selects the Nth distinct or Nth distinct index parameter respectively.

 

Added in Q4 22 - Improved mapping capabilities

Territories Wizard

  • The Territories wizard (previously used with Microsoft MapPoint) has been repurposed to work with Microsoft Bing Maps, allowing you to more easily generate map visualisations that are thematically shaded, either according to the number of people, or another selected metric. Setting up shape files is simpler, and the types of variables you can use extend beyond the typical options of Postal Sector, District and Area to include, for example, political or health region boundaries and even custom territories.

    Using the Territories wizard, you have multiple ways of creating virtual variables that are associated with map shape files and used to create the visualisations.

    See Territories Wizard: Overview

Drive Zone Wizard

Bing Driving (Isochrone API)

  • The default algorithm for generating drive time selections in the Map tool is now the Bing Driving (Isochrone API). The former Routes API algorithm remains as a Travel Type option in the Drive Time Settings panel.

 

Added in Q4 22 - Modelling developments

Behavioural Modelling

This release extends the functionality for visualising and identifying a diverse set of predictive features for use in a data model and includes:

  • Classification of features into positive, negative and balanced 'insight types'

  • Further chart options for the selection and display of dimensions

See Behavioural Modelling Development Update - Q4 2022

 

Standard Modelling

The concepts, charts and measures described for behavioural modelling are also fully supported for standard modelling.

See Standard Modelling Development Update - Q4 2022

 

Performance improvements

The processing of dimension results is now parallelized to improve performance. The three main stages are:

  • Caching the base and analysis selections - used in the profiling of all dimensions

  • Evaluating the Point-in-Time expression - each person’s event timepoint to aggregate their behaviour

  • Running the profiles for each dimension - in batches of 10 by default

See Parallel Running of Dimensions

 

Added in Q4 22 - new Expression functions

  • A new CreateTextList function has been added which allows you to create a text list from a number of strings

  • The Index (Logical) function has been extended to take text list parameters

 

Added in Q3 22 - Modelling Environment enhancements

Developments include:

 

Added in Q3 22 - improvements for drivetime calculations created through Point to Point wizard

  • Introduction of the Bing API isochrone means that systems with latitude and longitude variables marked with the GeoFormat property can benefit from significant drivetime improvements. You can now use the Point to Point wizard to process larger volumes of records and provide a user-specified level of accuracy in drivetime calculations.

    For the simplest case, you must specify the maximum drivetime (for example 0-80 minutes) and accuracy (for example to within 5 minutes) and the resulting variable will be a numeric with values of 5, 10, ..., 75, 80 - or a missing value if further away than 80 minutes.

    You can also specify the drivetime to be a walking time calculation.

    See Point to Point Wizard: Overview

 

Added in Q3 22 - Drive Zone wizard to use Bing Isochrone API

  • The Drive Zone wizard will use Bing Isochrone API in lieu of the former Routes API. Drivetimes return significantly more quickly and have more accurate zones, with approximately ten times as many vertices on the drivetime shape. The option to construct zones using walking time is also available.

    See Drive Zone Wizard: Overview

 

Added in Q3 22 - new extensions to Pattern Match aggregation capability

  • Two new extensions have been added:

    • You can specify an include or exclude list of codes from the pattern variable which are either the only ones of interest (include list) or ones that you do not want to see (exclude list).

    • In addition to first and last, you can return an intermediate, Nth element of a pattern

 

Added in Q3 22 - new Expression function

  • A new LogN function has been added which takes the logarithm of the number to the given base.

    You can find this with Expression: Maths Functions

 

Added in Q3 22 - updated Expression function

  • In Location Functions, the GeoPointInArea expression benefits from the following updates:

    • a significant increase in speed

    • its first two parameters can be variables, but all other parameters must be fixed numeric values

    See Expression: Location Functions

 

Added in Q2 22 - Modelling Environment enhancements

  • A new metric - Uplift PWE - provides an alternative to the pre-existing "Power" metric and can be used, along with a selection of new charts, to help you identify and select the best behavioural features to use in your data model.

    See Identifying the best Behavioural Features to use in a model

  • The chart previously called Analysis % has been renamed Analysis Rate to avoid confusion with Percent of Analysis. For each category, the analysis rate shows the proportion of people in the analysis selection and, so, is equivalent to "response rate" in a scenario where the analysis selection is the responders.

 

Added in Q2 22 - improved DriveTime calculations in Map tool

  • With Bing's introduction of a native Isochrone API endpoint, it is possible to calculate drivetimes on maps more accurately, and often faster, compared to the original FastStats' drivetime algorithm. The new method also provides options for calculating walking and public transport travel types.

    See Bing Maps: DriveTime Calculations

 

Added in Q2 22 - new Date/DateTime bandings on Cubes

  • When you add a Date or DateTime variable as a cube dimension, there are four new banding options available via the right-click context menu:

    • Day of Month which relates to both variable types

    • Hour and Minute of Day, Minute of Hour and Second of Minutefor use with a DateTime variable

  • For more information see Cube: How do I create a Cube dimension with Date or DateTime Variables?

 

Added in Q2 22 - ability to add Date/DateTime Expressions as a Cube dimensions

  • You can drag a date or datetime expression directly onto a cube as a dimension without the need to create a virtual variable; you can then band these dimensions using any of the right-click options available for date or datetime variables, with prompts to define start and end years for the expression, if the default current year +/- 5 years is not appropriate.

    See Cube: How do I create a Cube dimension with a Date or DateTime Expression?

 

Added in Q2 22 - use of Regular Expressions (Regex) with new String expression functions

 

Added in Q2 22 - new Date and Time expression functions

  • Two new functions have been added to work with Date and DateTime expressions and tell you if the given date/datetime is a weekday or weekend.

    • IsWeekday - returns whether the given date/datetime is a weekday.

    • IsWeekend - returns whether the given date/datetime is a weekend.

 

Added in Q1 22 - Segmentation Migration extensions

  • The Migration (2-points) and Migration (All points) segmentation reports have been extended to include simplified general and specific views of people's movement between segments. You also have the option to display the reports in terms of volume only, or volume and value.

    See the Migration (2 points) and Migration (All points) topics for full details.

 

Added in Q122 - Modelling Environment enhancements

  • Data Grid options provide the flexibility to display values associated with evaluation contexts as well as training contexts in a Data Grid

  • Improved context evaluation

    • The default approach used when evaluating behavioural features has changed to provide a stronger indication of the effectiveness of predicting behaviour during the evaluation stage of the process.

  • Improved Decision Tree integration within the Modelling Environment

    • The Results tab of the Modelling Environment captures and lists models generated using the Dimensions you choose to explore. Previously this was only possible for Decision Trees built without any manual intervention - that is, by simply clicking the Build button. You can now capture Decision Trees built and modified using the manual options, such as growing or pruning particular branches.

  • Enhanced Decision Tree performance

    • When you create Behavioural Features within the Modelling Environment, there is extensive use of cached variables and expressions, often nesting several levels deep within selections. There are updates and improvements to the long-standing mechanisms used within FastStats for creating requests and evaluating the cache. As well as particularly noticeable when using Decision Trees with Behavioural Features, these changes benefit many areas of FastStats and have been patched back to the Q4 21 release.

  • Improved annotation of Selections, Dimensions and Results

    • When creating behaviour-based models, you can document your exploration and thought processes by editing descriptions and adding notes associated with Selections, Dimensions and Results generated as part of your modelling investigations.

  • Labelling enhancements

    • The automatic bands for the “Time since...” Behavioural Feature have been adjusted. For example, the “number of months since” feature breaks down the first 12 months and then groups any longer time-frames together. Because a "Time since..." feature starts counting at 0, only months 0 to 11 are broken down individually.

    • The wording on the “Modify time frame” dialogue has been improved to clarify the interval created.

 

Added in Q1 22 - new Expression function

  • System Functions

    • VarCountsInfoFind - returns the index of the code or description to find for your given variable.

 

Added in Q1 22 - Map Wizard enhancements

  • Drive Zone Wizard

    • For all three methods of calculating zone size, you can create zones more accurately using latitude and longitude variables, instead of using a geographic variable which is accurate only to the level of the underlying geographic shape files.

    • An option for using Driving Distance to create the zones has been added.

  • Point to Point Wizard

    • When selecting to create a variable based upon straight line - as the crow flies - distance, you can specify latitude and longitude variables and have FastStats create an expression based on the centre point to calculate the distance.

    • For all options, you can specify a maximum value in terms of drive time, drive distance or straight line distance and only the records within that area are then categorised in the resulting virtual variable.

 

Added in Q4 21 - Modelling Environment enhancements

  • Two new Behavioural Features have been added:

    Count distinct

    Time since first

  • More flexible specification of time frame to restrict the transaction rows being analysed

  • A number of usability improvements have been made to the handling of Dimensions (Behavioural Features) in behavioural modelling, including:

    • More streamlined interface with dialog-based menus

    • Improvement to the way criteria are applied

    • More practical batch processing for handling large numbers of Dimensions

    Behavioural Modelling Development Update - Q4 2021 provides full details.

 

Added in Q4 21 - Aggregation Expressions

  • Running/Rolling - now also support a rolling time window allowing you to determine, for example, the highest amount spent by a customer in any 12 month period.

Added in Q4 21 - Expressions

  • Derived Cube Expressions

  • Selector Expressions

    • NthSelectedFlag - to return the description of the Nth selected flag in a flag array

    • NthIndexSelectedFlag - to return the index of the Nth selected flag in a flag array

  • Set Expressions

    • IsSubset/IsSuperset - now have an optional parameter to specify 'proper' subset/superset

  • System Expressions

    • VarCountsInfoNth (natural order for varcodes) - allows the expression to pull out information from the VarCodes for a variable

  • Postcode Expressions

    • File now uses ONS data file for improved and more up-to-date postcode list

 

Added in Q3 21 - new list from file functions added to File Functions in Expressions

  • Two new File Functions allow you to refer to a column in a delimited data file and create a numeric or text list data type which you can use as parameters within other Expression functions. Should the values in your list need to change, an update to the data file will apply to the Expression itself, and also to any saved analyses in which that Expression has been used.

    Expression: File Functions

 

Added in Q3 21 - Expressions with open-ended parameters extended to allow new list from file parameters

  • Making the list from file functions widely usable, 26 existing functions with open-ended parameters have been extended to also allow list parameters. These are:

    Geo: GeoNearest; GeoDistMin; GeoNth; GeoDistNth; GeoPointInArea

    UKPostcode: UKPostcodeNearest; UKPostcodeDistMin; UKPostcodeNth; UKPostcodeDistNth

    String: AddStr; StrContains; StrContainsl; StrBegins; StrBeginsl; StrEnds; StrEndsl; StrShredContains

    Other: VarSelect; InList; Index; MaxIndex; MinIndex; BandUp; BandDown; Nth; NthIndex

  • Existing utility functions on lists (CountList, StrList) have also been extended to take Numeric or TextList parameters.

  • A new function – TextListContains({textList}, string, string…) has been created to search a text list for the specified values and return the index of the first match found.

 

Added in Q3 21 - Modelling Environment enhancements

 

Added in Q3 21 - the ability to update Segmentation reports using a scheduled task

 

Added in Q2 21 - Event-driven behavioural modelling extended to allow for detailed customer journey selections

  • The ability to carry out simple, event-driven behavioural modelling has been extended and you can now use the FastStats Modelling Environment to define detailed customer journey – or ‘event-driven’ - selections that allow you to better understand the differences between your customers in terms of their previous behaviour.

    Overview of Customer Journey Selections

 

Added in Q2 21 - Improved thematics in Maps

  • You can now shade, colour and examine geographic distribution in a Map using any of the thematic types available in the Cube and Tree tools.

    Bing Maps: Shaded Map

  • Having gained your insight, you can use the new selectable Map Legend options to highlight and create a drag-off selection of any bands of interest.

    Bing Maps: Selectable Thematic Map Legend

  • For multiple similar layers on a Map, you can now right-click a Map Layer tab and copy before then editing the settings as required.

    Bing Maps: Copy Map Layer

 

Added in Q2 21 - new 'User Defined' thematic option added for Cube, Tree and Map tools

  • A new - User Defined - thematic type is now available in the Cube, Tree and Map tools.

 

Added in Q2 21 - Segmentation enhancements

 

Added in Q2 21 - Improvements to the Create from Cube wizard

  • You can now right-click and edit a Virtual Variable created using the Create from Cube wizard

  • You can now right-click and view a copy of the Cube or Tree used to generate the Virtual Variable

    Create from Cube Wizard: Overview

 

Added in Q2 21 - new Expression functions added

  • Seven new Expressions have been added in the Q2 release.  This is to support development within Elapsed Time Segmentation and also to add consistency and power to DateTime analysis.

    • Age functions - AgeHours, AgeMinutes, AgeSeconds

    • Date functions - DateTimeStart, DateTimeEnd, FinancialDateTimeStart, FinancialDateTimeEnd

  • The following existing functions have been extended:

    • Case/And/Or expressions - now allow up to 65k parameters

    • MakeDateTime(int) function - alterered to also accept one integer parameter

 

Added in Q1 21 - new relative points and offsets provide more capabilities when working with DateTime variables

  • New functionality has been added to the DateRule dialog when working with DateTime variables. It is now possible to find information in almost real-time, e.g. Find me all the people who have booked in the last 4 hours...Select people who have booked a holiday so far today.

    Using Variables In Selections: DateTime Variables

 

Added in Q1 21 - Decision Tree enhancements

 

Added in Q1 21 - Map drag-offs improved to accurately select people within the shapes drawn on the map