yo @microsoft/sharepoint fails for SPFx Dev Preview in VIsual Studio Code

The Hello World tutorial which describes the use of the Yeoman template to create a SharePoint Framework web part in Visual Studio Code suggests that the template should be installed globally.

When I follow these instructions, I am unable to create the project using yo @microsoft/sharepoint - that call gives the message:

Error @microsoft/sharepoint
You don't seem to have a generator with the name @microsoft/sharepoint installed....

I have found that installing the template locally (without the --g parameter) allows me to call the template

The Common Data Model - yet another back-end data store?

Recently integrated into the PowerApps preview (used to be called a beta, I believe?) is an option on the left navigation termed "Entities". Click on that and a new Common Data Model ("CDM") database will be created for you.

This is a new type of business database being offered by Microsoft for use in delivering business applications (see https://businessplatform.microsoft.com for an introduction to this initiative). Currently its purpose in the mix of cloud services is to act as a data store for apps built with PowerApps, Power BI and Flow. It comes pre-configured with commonly-used data types known as Entities, such as "customer" and "purchase order". These entities each contain sets of fields for relevant data. The PowerApps web studio lists all these entities, and offers a link to interact with each entity in Excel - this allows data in the entity to be viewed and updated. This pre-configuration, and the ability to manage these entities, effectively places the CDM one abstraction level above SQL Server.

Seems as though SharePoint is destined to become a second-class data store when it comes to PowerApps. But that's not such a Bad Thing really - SharePoint lists were never intended to be used to model and contain relational data. I like the idea that these new entities and the Common Data Model hide the management of columns, tables, views and stored procedures which are necessary when using SQL Server. The original announcement discusses the fact that the CDM is built upon Azure key capabilities, so scaling (and hopefully performance) are ensured out of the box.

Losing control over data access methods could be a cause for concern for developers, as could the potential for pre-configured data relationships which may not represent a company's view of the world. But the idea of allowing organizations to replace the mix of Excel spreadsheets, Access databases and other sundry data stores that often contain key business data & processes has got to be worth considering! It's going to be interesting to see whether this latest concept has legs....

Expose an ES2015 Module as a Global with Browserify (react-addons-update)

Calling the React setState() method within TypeScript in Visual Studio Code requires that all properties of the state object are set in the parameter to that method. That was proving a nuisance when there are a few state properties, so I decided to use the update addon available in React - that allows for a much simpler command such as

let newState = update(this.state, {
    message: {$set: "Success!"}
});
this.setState(newState);

Adding the react-addons-update module without any other changes to the build operations in gulp caused the content of that addon to be included in the bundled JS when processed by browserify.

That additional content can be excluded from the bundled JS by these adjustments:

1. Use the "external()" method on browserify to remove it from the bundle

       let bundler =  browserify({
               entries: config.rootJS,
               debug: true //This provides sourcemapping
           })  //Initialising browserify
           .external(['react', 'react-addons-update', 'react-dom', 'react-router']);
   

2. Add a shim entry in the package.json file to map a global name to the addon

     "browserify-shim": {
       "react": "global:React",
       "react-addons-update": "global:React_Addons_Update",
       "react-dom": "global:ReactDOM",
       "react-router": "global:ReactRouter"
     }
   

3. Create an additional JavaScript file containing the ES2015 addon code transpiled to E5

That last stage required a new gulp task, passing the file "update.js" file from the React source to browserify.

However, that new bundled file for the addon did not expose a global name that could be used within my custom bundle. The key to exposing this global name proved to be adding the "standalone" parameter in the browserify call (line 6 in the code below).

gulp.task('packageReactAddons', () => {

    let bundler =  browserify({
            entries: 'node_modules/react/lib/update.js',
            debug: false,
            standalone: 'React_Addons_Update'
        });

    return bundler.bundle()
        .on('error', console.error.bind(console))
        .pipe(source('react_addons_update.js')) // Pass desired output file name to vinyl-source-stream
        .pipe(gulp.dest(config.librariesOutputPath)); // Destination for the bundle
})

After uploading the resulting JS file to a page, I could see the object named "React_Addons_Update" in the global name space of the page. Yeah!

Using a JavaScript Library in TypeScript Modules - the D.TS FIle

Update - After restructuring the JavaScript code to separate views from components, as per the great article by Brad Westfall, I found that the local custom declaration file no longer satisfied the compiler. The reason is that the views and container components are placed in separate folders, which breaks the relative module path references. For the sake of expediency, I have resorted to moving the custom declaration file into the SPScript node_module folder. Not great, I know, but it works and so will do in the meantime.

If you are needing to hand-craft your own d.ts files, have a look at Complex Typescript Definitions Made Easy by Steve Fenton. You'll be glad you did!

The JavaScript library SPScript simplifies REST calls to SharePoint from the client by wrapping the AJAX requests in higher level abstractions. Seemed a useful library for my trials with TypeScript, React and D3 in presenting SharePoint data, so I added it to my source code in Visual Studio Code, and imported the file into my module App.ts.

Issue - Cannot Find Module SPScript

Its not that simple in Typescript. First issue to correct is the red squiggly that appeared under SPScript in the following statement:

import SPScript from "SPScript";

This error notification from the editor environment indicates that TypeScript cannot locate a module that matches that import statement. Looking through the module resolution notes in the online TypeScript handbook shows the locations and files which the compiler will check for the sought module (see the section titled "How TypeScript resolves modules" on that page for the details). SPScript is provided as a plain JavaScript library, without .TS or .D.TS files. So none of the files expected by the node name resolution would be found.

To correct this, it was necessary to add a file at one of those locations. Rather than make any adjustments to the SPScript node module (risky!), I added the declaration file SPScript.d.ts in the same folder as the App.ts file, and added an export statement to that new file to set this file as a module.

A note on Visual Studio Code - I have noticed that VS Code is sometimes slow to indicate that errors are corrected. In this case, the red squiggly did not disappear, even with the new file in place. If you close and reopen the file with the issue (in my case, App.ts) then the error was gone when the file reopened!

Issue - What to Include in the Declaration File?

The call I wanted to make using SPScript is as follows:

    const dao = new SPScript.RestDao(_spPageContextInfo.webAbsoluteUrl);

For TypeScript to recognize the types in those statements, the declaration file SPScript.d.ts needs to define the various classes and parameters. I tried various combinations of statements in the declaration file, eventually using this:

export module SPScript {
    export class RestDao {
        constructor(webUrl: string);
    }
}

This approach allowed TypeScript to resolve the types, and the compiler successfully created a JS file.

So, all ready to go? Nope - looking at the code in the compiled JS file, that statement had become

    var SPScript_1 = (typeof window !== "undefined" ? window['SPScript'] : typeof global !== "undefined" ? global['SPScript'] : null);
    //... other statements
    var dao1 = new SPScript_1.SPScript.RestDao(_spPageContextInfo.webAbsoluteUrl);

That would fail in the browser, as the inclusion of the library SPScript.JS in the HTML page gives access to SPScript.RestDao not to SPScript.SPScript.RestDao.

Plenty of investigation, trial and error lead to the use of this in the declaration file:

export class RestDao {
    constructor(webUrl: string);
}

The import statement was adjusted to

import { RestDao } from "./SPScript";

And the successful outcome of the compile process gives this in the JavaScript file:

    var SPScript_1 = (typeof window !== "undefined" ? window['SPScript'] : typeof global !== "undefined" ? global['SPScript'] : null);
    //... other statements
    var dao = new SPScript_1.RestDao(_spPageContextInfo.webAbsoluteUrl);

A note on the Global Name and Browserify-Shim

The ES5 JavaScript file is being created from the TypeScript modules with the help of Browserify within a Gulp task

gulp.task('package', ['includeLibs', 'compile'], () => {
    let bundler =  browserify({
            entries: config.rootJS,
            debug: true //This provides sourcemapping
        })
        .external(['./SPScript']); 

    bundler.bundle() //start buindling
        .on('error', console.error.bind(console))
        .pipe(exorcist(config.distOutputPath + '/' + config.bundleFile + '.map')) //Extract the sourcemap to a separate file
        .pipe(source(config.bundleFile)) // Pass desired output file name to vinyl-source-stream
        .pipe(gulp.dest(config.distOutputPath)); // Destination for the bundle
})

Notice line 6 - this specifies that the SPScript is not to be included in any way in the output JavaScript. In addition, the global SPScript name needs to be made available within the bundled output modules. This is achieved by including the browserify-shim module in the source code, and adding the following to the package.json file:

  "browserify": {
    "transform": [
      "browserify-shim"
    ]
  },
  "browserify-shim": {
    "./SPScript": "global:SPScript"
  }

Note that the exact text used in the import statement - in this case it was ./SPScript - must be passed to browserify-shim, so that the correct reference occurs within the compiled JavaScript.

Modern JavaScript in SharePoint (4) - Gulp and PnP

The articles:

• Modern JavaScript in SharePoint (1) - the Motivation
• Modern JavaScript in SharePoint (2) - Visual Studio Code
• Modern JavaScript in SharePoint (3) - npm and gulp
• Modern JavaScript in SharePoint (4) - Gulp and PnP

PnP is the Office 365 Developer Patterns and Practices program - a (very) comprehensive resource illustrating techniques for coding against SharePoint and Office 365. Amongst the many offerings is the JavaScript Core Library. This library wraps the calls to the SharePoint REST API in a set of objects (or, to be accurate, in a series of namespaces and functions that mimic an object collection - this is ES5 JavaScript, after all!). The addition of this layer of abstraction simplifies many of the operations that are possible from client-side code.

For example, submitting a search against SharePoint becomes as simple in JavaScript as this:

pnp.sp.search({ Querytext: "Document", RowLimit: 5 })
 .then(function(data) { console.log(JSON.stringify(data)); })
 .catch(function(e) { console.log("Error: " + e.message); });

The calls in the library handle the asynchronous nature of the interactions - hence the "then" and catch" functions for processing the retrieved data or any error that occurs.

How can this library be incorporated into a build workflow using ES6/ES2015 JavaScript in Visual Studio Code? In my trials, I wanted to integrate the build and deployment process using gulp, so first I investigated options for upload files to SharePoint Online within a gulp task. And happily there is a gulp plugin ready to achieve that - its called gulp-spsave. It is very simple to use, with the only disadvantage being that the credentials used to connect to Office 365 are supplied as plain text within the call. An alternative is to use gulp-spsync by Wictor Wilen - but for the purposes of these tests I have stuck with the spsave plugin.

This allows the upload of files into the site assets library in SharePoint to be achieved by:

gulp.task('upload-sp', function () {
    return gulp.src(config.bundleFilePath)
        .pipe(print())
        .pipe(spsave({
            username: settings.username,
            password: settings.password,
            siteUrl: settings.siteUrl,
            folder: "SiteAssets",
            notification: true
        }));
})

In this task, the account details and the target site are stored in a separate settings JSON file. I also have a separate similar task to upload libraries, such as PnP and any other dependencies.

Compiling PnP in Typescript in Visual Studio Code

So, now to use PnP in JavaScript in Visual Studio Code. First step is to add the module into the code's folder structure using the npm call on the command line (in the useful integrated terminal window):

npm install --save-dev sp-pnp-js

This adds a folder into the node_modules directory tree that includes the pnp.js library.

Next, the PnP library needs to be imported into the JavaScript module in which calls will be made using the library. In ES6/ES2015 this is achieved by the line

import * as pnp from "sp-pnp-js";

That seemed simple. But its not quite that easy. When I added a PnP call in my code in Typescript, and attempted to compile it using the Typescript compiler, I received various errors. Visual Studio Code started giving several naming errors. The TypeScript compiler was looking into PnP.js and finding lots of names it did not recognize.

At run time in the browser, these names would be available in the global namespace as the PnP library is built knowing that various other JS libraries are always referenced from SharePoint pages. But the point of using strongly typed JavaScript is to check its validity at compile time, so the compiler needs to know about all names used in all the current scripts. So I needed to supply the compiler with references for all names used in PnP.js. Experimentation lead to the following inclusions for overcoming the compile errors:

• PnP is dependent on ES6 Promise and on whatwg fetch at runtime, so the type definitions for these libraries are needed. As an example, a compile error may complain of "promise" being an unknown name. These definitions can be added to the source code in VS Code via the following statements which get the necessary files from the DefinitelyTyped online resource:
  typings install dt~es6-promise --global --save
  typings install dt~whatwg-fetch --global --save
• PnP also makes considerable use of the Microsoft Ajax library. I could not find a command line statement with which to add this type, so ended up manually getting the Microsoft-ajax.d.ts file from a GitHub location. I then manually added a sub-folder under the typings globals folder structure, and included a reference to this in the index.d.ts file under typings.
• Last name correction was to add SharePoint-specific name declarations. For these, I found a file SharePoint.d.ts which I also manually added to the typings in this source.

This took a while, but finally managed to get the Typescript compiler to accept PnP!

Bundling The JavaScript for Deployment

Once the code could be compiled into an ES5 file, the ES5 then was ready for use on a test SharePoint page. Or was it? Looking at the file created using browserify in a gulp task, it was a huge file, as it included all of the PnP library. Clearly this is not what we want (as the PnP.js minimized library would be referenced from the page), so needed to tell browserify to exclude that library. This is done using the "external()" command:

    return browserify({
            entries: config.sourceJS + 'spTests.js',
            debug: true //This provides sourcemapping
        })
        .external(['sp-pnp-js'])
        .bundle()
        .on('error', console.error.bind(console))
        .pipe(source(config.bundleFile)) // Define the name of the bundle
        .pipe(gulp.dest(config.tsOutputPath)); // Destination for the bundle

In this gulp task, the compiled file "sptests.js" is being prepared for upload - the reason I am taking this approach is to be able bundle multiple modules into a single file. The second command in the pipeline tells browserify not to include the pnp.js library in the bundled output. Great, the completed file is now much smaller.

Using the file in SharePoint

The bundled file is uploaded to SharePoint using gulp-spsave, along with the libraries on which it is dependent. It nearly ran in the test page, but showed an error - the name "pnp" was not recognized in the script available on the page. Looking at the PnP library, it exposes "$pnp" rather than "pnp". So the gulp process needed to somehow map all calls to pnp to actually use $pnp.
browserify-sim to the rescue. Adding the following to the package.json file, and running the gulp tasks again, gave a bundled ES5 file that successfully uses PnP calls!

  "browserify": {
    "transform": [
      "browserify-shim"
    ]
  },
  "browserify-shim": {
    "sp-pnp-js": "global:$pnp"
  }

Thoughts on the Process

As you can see, writing code in Visual Studio Code in Typescript against PnP and getting that code to run in a SharePoint page is at the moment a mission. I am sure it will get easier.... If you need any tips about the concepts here, please drop leave a comment & I'll be in touch.

Reflections on Learning React....

What's quiet amusing (and frustrating….) is that the simplest user interactions with page elements have to be relearnt in a way fitting the paradigms of the React model. As an example, imagine a search form. Just a text box and a button, simple right? In "classic" JS, adding an onclick event handler to read and search on the value in the textbox takes no thought at all (ignoring here the "production -quality" trimmings that are necessary in a real world form, such as validation).

Reproducing this interaction in React requires knowledge of a few principles:

• Event handlers are attached to an element as a property. If using JSX, be careful not to enclose the handler in quotes!

   < input onclick={this.submitSearch} type="button" value="Search" />
  

• The "this" context pointing to the React component is not available in the event handler. That means that if the value in the text box is bound to a member of the component state, this value would not be accessible in the handler. There are two approaches to correcting this - either add the "bind(this)" call to the end of the handler invocation

   < input onclick={this.submitSearch.bind(this)} type="button" value="Search" /> 

  or define the handler as an arrow function

  submitSearch = ( e ) => { … }

• Bind the text box default value to an initial value. Do not bind the value property, as doing so causes the box to behave as read only in the form. Also need to add an onchange handler for the text box, bound to a function that updates the appropriate member of the state object with the value from the control (e.target.value)
• Add the shouldComponentUpdate() lifecycle function to the component, ensuring that the change of text value in the state member does not trigger the update (unless you want the search to occur on every key stroke)
• Setting the state in React (using Typescript in VS Code) requires all members of the state object to be set each time

Browserify is Sensitive in its Entry Files Parameter - "Cannot Find Module" Error

Faced an issue today whilst using Browserify to bundle some JavaScript files. The JS files had been compiled in Visual Studio Code from Typescript TSX files via gulp, and were all stored together in a single intermediate folder.

The single JS output bundle file was created just fine by Browserify. But when used in an HTML page, that file would result in a "cannot fund module" error. This error is created in the _prelude.js module, part of the output from Browserify.

So, it was time to explore the contents of the bundled JavaScript file. That file contains sections for each of the bundled modules. The last of those modules is generally the one that executes after loading. If that module needs to reference methods or properties exported by other modules, then there should be a mapping between the exported module name and that module key in the bundled modules array. Looking at the end of my output file, that mapping was as follows:

},{"./AppHeader":undefined}]},{},[1,2,3,4])

This means that the "AppHeader" module was not reachable, leading to that "cannot find module" error. After plenty of research into the bundling tool (during which I came across this useful article by Ben Clinkinbeard that describes the Browserify actions), I went back to look at the calling task in gulp. There I found that the Entry Files parameter was using a glob to point at all the JS files in the intermediate folder - the result of some previous testing in a project on which this set of files was based.

I changed this parameter to point just at the single root JS file, reran the bundling task in gulp, and now the output file ended with this:

},{"./AppHeader":1}]},{},[2])

This new bundled file worked just fine in the browser. So beware the Entry Files parameter!

PS For lots more details on this tool, consult the Browserify handbook!